Alignment method, apparatus, electronic device and storage medium for a large language model

Abstract

The application provides an alignment method of a large language model. The method comprises the following steps: obtaining a pre-training large language model, and initializing the pre-training large language model into a text completion task state; performing model feature retrieval on a target data set to determine at least one target large language model; constructing a context alignment prompt word according to the target large language model, and inputting the context alignment prompt word into the pre-training large language model; obtaining target conversation text, updating the target conversation text into the context alignment prompt word, and reasoning to obtain conversation response text; and repeatedly performing the following steps: adding the conversation response text of a current dialogue round into the context alignment prompt word; obtaining target conversation text of a next dialogue round, inputting the target conversation text and the context alignment prompt word into the large language model, obtaining conversation response text of the next dialogue round, and setting the next dialogue round as the current dialogue round. The method can realize efficient alignment of the pre-training large language model.

Description

Technical Field

[0001] This application relates to the field of large language models, and more particularly to an alignment method, apparatus, electronic device, and storage medium for large language models. Background Technology

[0002] Large Language Models (LLMs) are a class of deep learning models capable of understanding and generating natural language text. They have shown great potential in fields such as language understanding, text generation, and question answering systems. These models learn through large-scale text data during the pre-training stage and are then fine-tuned on specific tasks to achieve more accurate and specific language understanding and generation. Large Language Model Alignment is the process that guides large language models to learn human interaction preferences, follow human instructions, and avoid generating harmful text.

[0003] In related technologies, alignment methods typically rely on additional human data and fine-tuning training to achieve instruction following, such as Supervised Fine-Tune (SFT) and Reinforcement Learning from Human Feedback (RLHF). These methods require significant human intervention and tedious training processes, resulting in high model alignment costs and low efficiency. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for aligning large language models, which can achieve efficient alignment of pre-trained large language models without relying on a large amount of manual data.

[0005] To achieve the above objectives, a first aspect of this application proposes an alignment method for a large language model, the method comprising:

[0006] Obtain a pre-trained large language model and initialize the pre-trained large language model to enter the text completion task state; wherein, the dataset corresponding to the pre-trained large language model during pre-training is the target dataset;

[0007] Model feature retrieval is performed on the target dataset to identify at least one target large language model;

[0008] Context alignment prompts are constructed based on the target large language model, and the context alignment prompts are input into the pre-trained large language model so that the pre-trained large language model aligns with the target large language model.

[0009] Obtain the target conversation text and add it to the context-aligned prompt words so that the pre-trained large language model can perform task processing on the target conversation text to obtain the conversation response text;

[0010] Repeat the following steps:

[0011] Add the conversation response text of the current dialogue round to the context-aligned cue word;

[0012] In response to obtaining the target conversation text for the next conversation round, the target conversation text for the next conversation round and the context alignment cue words are input into the pre-trained large language model to obtain the conversation response text for the next conversation round, and the next conversation round is set as the current conversation round.

[0013] In some embodiments, the context-aligned cue word includes a first cue word field, and the construction of the context-aligned cue word based on the target large language model includes:

[0014] Get target keywords, preset layout format, and preset dialogue structure format;

[0015] The format constraint field is generated based on the preset layout format and the preset dialogue structure format;

[0016] Based on the target name of the target language model, the target keywords, the layout format, and the dialogue structure format, a first prompt word field is generated;

[0017] Based on the first prompt word field, construct the context-aligned prompt word.

[0018] In some embodiments, the context-aligned prompt word further includes a second prompt word field, and the construction of the context-aligned prompt word based on the target large language model includes:

[0019] Obtain the target name and target system words of the target large language model;

[0020] The second prompt word field is constructed based on the target name and the target system term;

[0021] The context-aligned prompt word is constructed based on the first prompt word field and the second prompt word field.

[0022] In some embodiments, the context-aligned prompt word further includes a third prompt word field, and the construction of the context-aligned prompt word based on the target large language model includes:

[0023] Obtain the first target safe word of the target large language model; wherein, the first target safe word is used to guide the large language model to generate safe text;

[0024] The third prompt word field is generated based on the first target security word;

[0025] The context-aligned prompt words are constructed based on the first prompt word field, the second prompt word field, and the third prompt word field.

[0026] In some embodiments, the context-aligned prompt word further includes a fourth prompt word field, and the construction of the context-aligned prompt word based on the target large language model includes:

[0027] Obtain the identity inquiry text, and based on the identity inquiry text, obtain the actual identity response text of the target large language model for the identity inquiry text;

[0028] The identity constraint field is constructed based on the identity inquiry text and the actual identity response text.

[0029] The context-aligned prompt words are constructed based on the first prompt word field, the second prompt word field, the third prompt word field, and the fourth prompt word field.

[0030] In some embodiments, the context-aligned prompt word further includes a fifth prompt word field, and the construction of the context-aligned prompt word based on the target large language model includes:

[0031] Obtain the capability query text, and based on the capability query text, obtain the actual capability response text of the target large language model for the capability query text;

[0032] The capability constraint field is constructed based on the capability query text and the actual capability response text.

[0033] The context-aligned prompt words are constructed based on the first prompt word field, the second prompt word field, the third prompt word field, the fourth prompt word field, and the fifth prompt word field.

[0034] In some embodiments, constructing the context-aligned cue word based on the first cue word field, the second cue word field, the third cue word field, the fourth cue word field, and the fifth cue word field includes:

[0035] Based on the preset layout format and the preset dialogue structure format, the context-aligned prompt words are generated according to the first prompt word field, the second prompt word field, the third prompt word field, the fourth prompt word field, and the fifth prompt word field.

[0036] To achieve the above objectives, a second aspect of this application provides an alignment device for a large language model, comprising:

[0037] The first acquisition module is used to acquire a pre-trained large language model and initialize the pre-trained large language model to enter the text completion task state; wherein, the dataset corresponding to the pre-trained large language model during pre-training is the target dataset;

[0038] The determination module is used to perform model feature retrieval on the target dataset and determine at least one target large language model.

[0039] The construction module is used to construct context alignment prompts based on the target large language model, and input the context alignment prompts into the pre-trained large language model so that the pre-trained large language model aligns with the target large language model.

[0040] The second acquisition module is used to acquire the target conversation text and add the target conversation text to the context-aligned prompt words so that the pre-trained large language model can perform task processing on the target conversation text to obtain the conversation response text.

[0041] Add a module to add the conversation response text of the current dialogue round to the context-aligned cue word;

[0042] The response module is used to respond to obtaining the target conversation text of the next conversation round, inputting the target conversation text of the next conversation round and the context alignment prompt words into the pre-trained large language model to obtain the conversation response text of the next conversation round, and setting the next conversation round as the current conversation round.

[0043] To achieve the above objectives, a third aspect of this application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the large language model alignment method as described in the first aspect.

[0044] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the large language model alignment method as described in the first aspect.

[0045] This application proposes a method, electronic device, and storage medium for aligning a large language model. The method includes acquiring a pre-trained large language model and initializing it to enter a text completion task state. The dataset corresponding to the pre-trained large language model during pre-training is the target dataset. Model feature retrieval is performed on the target dataset to determine at least one target large language model. Context alignment prompts are constructed based on the target large language model and input into the pre-trained large language model so that the pre-trained large language model aligns with the target large language model. Target conversation text is acquired and added to the context alignment prompts so that the pre-trained large language model processes the target conversation text to obtain conversation response text. The following steps are executed iteratively: adding the conversation response text of the current conversation round to the context alignment prompts; in response to acquiring the target conversation text of the next conversation round, inputting the target conversation text of the next conversation round and the context alignment prompts into the pre-trained large language model to obtain the conversation response text of the next conversation round, and setting the next conversation round as the current conversation round.

[0046] In contrast to the problems in the background art, the large language model alignment method provided in this application utilizes prompting engineering techniques. It constructs context alignment prompts by referencing the characteristics of the target large language model and leveraging the text continuation reasoning capabilities of a pre-trained large language model. These prompts are then input into the original pre-trained large language model, effectively guiding and aligning its behavior. This enables the pre-trained large language model, which originally only possessed text completion capabilities, to exhibit advanced capabilities similar to the target large language model, such as instruction following and task processing. Compared to traditional model fine-tuning training methods, this method avoids the high human and computational costs of constructing new instruction following datasets and fine-tuning training models. It requires no additional dedicated manual data or fine-tuning training, reducing alignment costs, improving alignment performance, and achieving rapid alignment compared to existing large language model alignment methods.

[0047] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0048] Figure 1 A flowchart illustrating an alignment method for a large language model provided in an embodiment of this application;

[0049] Figure 2 Provided for an embodiment of this application Figure 1 Flowchart for step 103;

[0050] Figure 3 Provided for another embodiment of this application Figure 1 Flowchart for step 103;

[0051] Figure 4 Provided for another embodiment of this application Figure 1 Flowchart for step 103;

[0052] Figure 5 Provided for another embodiment of this application Figure 1 Flowchart for step 103;

[0053] Figure 6 Provided for another embodiment of this application Figure 1 Flowchart for step 103;

[0054] Figure 7 A schematic diagram of prompt words provided for an embodiment of this application;

[0055] Figure 8 A schematic diagram of an alignment device for a large language model provided in an embodiment of this application;

[0056] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0060] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0061] Large Language Models (LLMs) are machine learning models with a large number of parameters and complex computational structures. Typically built from deep neural networks, they possess billions or even hundreds of billions of parameters, enabling them to handle more complex tasks and data. They exhibit strong generalization capabilities and can make accurate predictions on unseen data. LLMs have wide applications in Natural Language Processing (NLP), including but not limited to text generation, translation, summarization, and question answering systems. They can be trained on large-scale corpora to learn various grammatical, semantic, and contextual rules of natural language.

[0062] Base Large Language Models (Base LLMs) are pre-trained large language models that have not been fine-tuned for specific tasks. They are trained on massive amounts of text to predict the next word or sentence in a text sequence, but these predictions are not necessarily responses to specific instructions or dialogues. These models are the starting point for large language models (LLMs) and typically have broad language understanding and generation capabilities.

[0063] Supervised Fine-Tune Large Language Models (SFT LLMs) refer to further training a model using labeled data on top of its pre-trained version, making it more suitable for a specific task or domain. This process is called Supervised Fine-Tune (SFT), and its purpose is to guide the model to learn the specific rules required by the business and to stabilize the output format based on the knowledge learned in pre-training.

[0064] Reinforcement Learning Large Language Models (RLLMs) combine reinforcement learning (RL) techniques to learn policies through interaction with the environment, thereby optimizing the model's decision-making process. In the context of large language models, RL can be used to adjust the model's behavior to better align with human feedback and preferences.

[0065] With the powerful general-purpose performance of instruction-following language models such as ChatGPT and Claude, large language models have become a research hotspot in the field of artificial intelligence. LLMs are a class of deep learning models capable of understanding and generating natural language text, demonstrating enormous potential in areas such as language understanding, text generation, and question-answering systems. These models learn through large-scale text data during the pre-training phase and then fine-tune on specific tasks to achieve more accurate and specific language understanding and generation. Large language model alignment guides large language models to learn human interaction preferences, follow human instructions, and avoid generating harmful text. Existing alignment methods typically rely on additional human data and fine-tuning training to achieve instruction following, such as Supervised Fine-Tunneling (SFT) and Reinforcement Learning from Human Feedback (RLHF). These methods require significant human intervention and tedious training processes, leading to high economic and time costs. Furthermore, existing context alignment methods suffer from topic bias, which affects the accuracy and generalization of alignment, limiting their reliability in practical applications. In summary, existing alignment methods still have room for improvement in terms of effectiveness and efficiency, and suffer from problems such as reliance on additional data and fine-tuning training, high cost, and topic bias. Therefore, it is necessary to propose a more efficient, faster, more accurate, and more reliable alignment method to meet the needs of large language model alignment in more practical applications.

[0066] This application proposes a method, electronic device, and storage medium for aligning a large language model. The method includes acquiring a pre-trained large language model and initializing it to enter a text completion task state. The dataset corresponding to the pre-trained large language model during pre-training is the target dataset. Model feature retrieval is performed on the target dataset to determine at least one target large language model. Context alignment prompts are constructed based on the target large language model and input into the pre-trained large language model so that the pre-trained large language model aligns with the target large language model. Target conversation text is acquired and added to the context alignment prompts so that the pre-trained large language model processes the target conversation text to obtain conversation response text. The following steps are executed iteratively: adding the conversation response text of the current conversation round to the context alignment prompts; in response to acquiring the target conversation text of the next conversation round, inputting the target conversation text of the next conversation round and the context alignment prompts into the pre-trained large language model to obtain the conversation response text of the next conversation round, and setting the next conversation round as the current conversation round.

[0067] In contrast to the problems in the background art, the large language model alignment method provided in this application utilizes prompting engineering techniques. It constructs context alignment prompts by referencing the characteristics of the target large language model and leveraging the text continuation reasoning capabilities of a pre-trained large language model. These prompts are then input into the original pre-trained large language model, effectively guiding and aligning its behavior. This enables the pre-trained large language model, which originally only possessed text completion capabilities, to exhibit advanced capabilities similar to the target large language model, such as instruction following and task processing. Compared to traditional model fine-tuning training methods, this method avoids the high human and computational costs of constructing new instruction following datasets and fine-tuning training models. It requires no additional dedicated manual data or fine-tuning training, reducing alignment costs, improving alignment performance, and achieving rapid alignment compared to existing large language model alignment methods.

[0068] The following will further describe a method for aligning a large language model, an electronic device, and a storage medium provided in the embodiments of this application. First, the method for aligning a large language model in the embodiments of this application will be described in detail.

[0069] Reference Figure 1 This is an optional flowchart of the alignment method for a large language model provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 101 to 106. It is also understood that this embodiment... Figure 1 The order of steps 101 to 106 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0070] Step 101: Obtain the pre-trained large language model and initialize the large language model to enter the text completion task state.

[0071] The pre-trained large language model learns extensive natural language knowledge during the pre-training phase, but its core capability is limited to predicting and generating the next reasonable word or lexical sequence based on the given context text—that is, text completion. Therefore, the purpose of step 101 is to place the pre-trained model in its native, most basic text generation mode, preparing it for the subsequent alignment process. It should be noted that although it only possesses text completion capabilities, the pre-trained large language model has acquired a certain degree of reasoning ability through learning contextual semantics. This potential semantic understanding and generation ability will be guided and aligned in subsequent steps through carefully designed prompts, enabling the model to exhibit advanced language interaction behavior similar to the target large language model.

[0072] Therefore, step 101, as the starting point of the alignment method, is important because it obtains a powerful but untrained basic large language model, which is a key prerequisite for achieving alignment without fine-tuning. Through initialization, the model is awakened to its initial text generation state, fully preparing it for the subsequent fine-tuning alignment process.

[0073] Step 102: Perform large language model feature retrieval on the target dataset to identify at least one target large language model.

[0074] The target large language model is a large language model that performs well in the target dataset, possessing advanced capabilities such as good instruction following and task processing. Integrating the names and features of these target large language models into the construction of alignment prompts effectively guides the pre-trained large language model to align with the behavior of the target large language model. Specifically, taking ChatGPT as an example, instilling the concept "I am ChatGPT" into the pre-trained large language model can awaken its knowledge related to ChatGPT seen in the pre-training data, enabling it to reproduce and simulate ChatGPT's instruction understanding, response generation, and other behavioral patterns as closely as possible. By utilizing the rich knowledge learned by the large language model during the pre-training stage, a knowledge foundation is laid for the subsequent design of alignment prompts. Through flexible prompt engineering, the potential semantic understanding and generation capabilities of the pre-trained large language model can be fully explored and stimulated, avoiding the high computational and human costs of existing large language model alignment fine-tuning methods.

[0075] In some embodiments, the most frequently mentioned, or most widely discussed, large language model can be retrieved from the target dataset as the target large language model, such as ChatGPT. Since large language models are probabilistic models, the more widely known the knowledge, the better it can learn. Therefore, using an existing AI name here is to make it think it is ChatGPT, and to better refer to the knowledge it has learned in the pre-training dataset related to AI assistant command following.

[0076] Step 103: Construct context alignment prompts based on the target large language model, and input the context alignment prompts into the large language model so that the pre-trained large language model can align with the target large language model.

[0077] The core of this method lies in leveraging the reasoning capabilities demonstrated by a pre-trained large language model in text completion tasks to establish appropriate textual contexts, enabling the large language model to generate subsequent aligned conversational text more reasonably. Therefore, the core idea of ​​this invention is to construct, as far as possible, textual contexts of the already aligned large language model's behavioral patterns, thereby activating the behavioral patterns of mature large language models (such as ChatGPT) that the pre-trained large language model has already mastered during pre-training. These cue words are carefully designed text fragments that guide the pre-trained large language model to understand and adapt to the behavioral patterns of the target large language model. The process of constructing cue words includes defining the dialogue format, setting roles, introducing conditions, system settings, security statements, role confirmation, and skill confirmation. These cue words are then input into the pre-trained large language model, allowing the model to learn and align within the context.

[0078] Reference Figure 2 In some embodiments, the context-aligned prompt word includes a first prompt word field, and step 103 may include, but is not limited to, steps 201 to 204.

[0079] Step 201: Obtain the target keywords, preset layout format, and preset dialogue structure format.

[0080] In step 201, the target keywords, preset formatting, and preset dialogue structure format are first obtained. For example, for a pre-trained large language model, the target keyword is determined to be "correct." These keywords represent the core capabilities and characteristics of the target model, enabling it to generate positive responses. The preset formatting can specify the use of ChatML or Markdown formats to organize the dialogue content, while the dialogue structure format defines the start and end markers of the dialogue, such as using specific symbols or codes to identify the speaker and the content being spoken.

[0081] Step 202: Generate format constraint fields based on the preset layout format and preset dialog structure format.

[0082] In step 202, the format constraint field ensures the structural consistency of the prompt words, enabling the pre-trained large language model to more easily recognize and process the information in the prompt words and generate well-formatted text.

[0083] Step 203: Generate the first prompt word field based on the target name, target keywords, layout format, and dialogue structure format of the target large language model.

[0084] In step 203, the target name provides a reference object for alignment for the pre-trained large language model, while the target keywords further refine the specific direction and content of the alignment. For example, a text paragraph is created that begins by declaring the format and structure of the dialogue, then introduces the role of ChatGPT and gives some positive descriptions of it, such as using natural language to describe the keywords and format of the subsequent text to be completed, "The following is the **correct** dialogue between ChatGPT (**GPT4**) and the user, organized in **ChatML** and **Markdown** formats:", where the dialogue start and end marks are "<|im_start|>" and "<|im_end|>", respectively.

[0085] Step 204: Construct context-aligned prompts based on the first prompt field.

[0086] In this step, the generated format constraint fields are added to the previously constructed context alignment prompts, forming the context alignment prompts. By adding format constraint fields, the format and structure of the context alignment prompts can be precisely constrained and standardized. This helps ensure that the response text generated by the large language model strictly follows the syntax rules of ChatML and Markdown, improving the accuracy and consistency of alignment.

[0087] Reference Figure 3 In some embodiments, the context-aligned prompt word also includes a second prompt word field, and step 103 may include, but is not limited to, steps 301 to 303.

[0088] Step 301: Obtain the target name and target system words of the target large language model.

[0089] The target name typically refers to the name of the well-known large language model being aligned, such as ChatGPT. The target system word, on the other hand, directly uses the native system prompt word adopted by the large language model in practical applications, such as "You are ChatGPT, a large language model trained by OpenAI based on the GPT-4 architecture".

[0090] Step 302: Construct a second prompt word field based on the target name and target system words.

[0091] Among them, a name constraint field can be constructed by combining natural language with the target name. This field is designed to give the pre-trained large language model a clear ChatGPT identity awareness.

[0092] For example, the target name, such as ChatGPT, is constructed into the string "<|im_start|>ChatGPT" according to a preset format and added to each round of dialogue in the context prompt words so that the target name is always emphasized in the context as the AI ​​assistant role in the conversation.

[0093] Specifically, the native system prompt word of the target large language model obtained in step 301, "You are ChatGPT, a large language model trained by OpenAI based on the GPT-4 architecture," can be directly used as the system word constraint field. It contains core attribute descriptions of ChatGPT, such as its identity, capabilities, and technical architecture.

[0094] The two fields mentioned above can be combined to form a second cue word field. This cue word simultaneously incorporates the model's "ChatGPT identity" and "native system cue" identification. It can effectively guide pre-trained large language models to mimic and reproduce ChatGPT's semantic understanding, response generation, and other behaviors.

[0095] Step 303: Construct context-aligned prompts based on the first prompt field and the second prompt field.

[0096] In this way, the context alignment prompts make the most of the key feature descriptions of the target model itself, including identity and system state. Under the guidance of these prompts, the pre-trained large language model can achieve preliminary alignment with the instruction-following text continuation ability of the target large language model.

[0097] Reference Figure 4 In some embodiments, the context-aligned prompt word also includes a third prompt word field, and step 103 may include, but is not limited to, steps 401 to 403.

[0098] Step 401: Obtain the first target safe word of the target large language model.

[0099] This includes retrieving the first target safety word set by the target language model itself. These are likely keywords used by the target model in its safety declaration field to constrain itself from generating dangerous, inappropriate, or harmful content. For example, words like "harmless."

[0100] In some embodiments, a second target safety word can also be obtained. This second target safety word is a high-frequency safety warning word displayed in other aligned large language model data besides the target large language model in the target dataset. For example, words or phrases such as "truthful and credible," "neutral and objective," "factual basis," and "avoid offensive language" may frequently appear in these aligned large language model data. These words reflect the safety and quality principles followed by these aligned large language models, and these high-frequency keywords or texts can be used as the second target safety words.

[0101] By combining this second target security word derived from the target dataset with the first target security word obtained from the target large language model system itself in step 401, a more comprehensive and practical security constraint field can be constructed.

[0102] Step 402: Generate a third prompt word field based on the first target security word;

[0103] After obtaining these two types of safety terms from different sources but with the same goal, step 403 merges them to generate a complete safety constraint field. This field can take the form of an imperative statement such as, "You are a useful, respectful, and honest assistant. Your answers must be socially responsible, therefore you can refuse to answer some controversial topics."

[0104] Step 403: Construct context-aligned prompts based on the first prompt field, the second prompt field, and the third prompt field.

[0105] Finally, in step 403, the security constraint field, i.e., the third cue word field, is combined with the other cue word fields constructed earlier to form a context-aligned cue word that includes format constraints, identity constraints, and security word constraints. This ensures that the cue words not only constrain the format and identity of the model's response text, but also impose clear restrictions on the security and quality of its output content.

[0106] Reference Figure 5 In some embodiments, the context-aligned cue word includes a fourth cue word field, and step 103 may include, but is not limited to, steps 501 to 503.

[0107] Step 501: Obtain the identity inquiry text, and based on the identity inquiry text, obtain the actual identity response text that matches the target large language model.

[0108] In this step, a set of predefined identity query texts is first obtained. These texts are questions used to explore the identity characteristics of the target large language model, such as "Who are you?", "What is your name?", and "What kind of AI assistant are you?". Then, each identity query text is input into the target large language model, and the corresponding response generated by the model is obtained—the actual identity response text. The actual identity response text reflects the target large language model's perception and expression of its own identity.

[0109] For example, if the target large language model is ChatGPT, its response to the identity inquiry text "Who are you?" might be "Hello! I'm ChatGPT. How can I help you?".

[0110] Step 502: Construct the fourth prompt word field based on the identity inquiry text and the actual identity reply text.

[0111] Specifically, the identity inquiry text and the corresponding actual identity response text can be combined into key-value pairs to construct the identity constraint field. These are then combined according to the format declared in the first prompt word field to form the complete identity constraint field.

[0112] Step 503: Construct context-aligned prompts based on the first prompt field, the second prompt field, the third prompt field, and the fourth prompt field.

[0113] In this step, the previously constructed prompt fields are combined to form context-aligned prompts. By constructing context-aligned prompts that include identity constraint fields, the identity features of the large language model can be aligned more comprehensively and accurately. When generating responses, the large language model will be influenced by the identity constraint fields, making it more consistent with the identity positioning and interaction style of the target large language model.

[0114] Reference Figure 6 In some embodiments, the context-aligned prompt includes a capability constraint field, and step 103 may include, but is not limited to, steps 601 to 603.

[0115] Step 601: Obtain the capability query text, and based on the capability query text, obtain the actual capability response text of the target large language model that matches the capability query text.

[0116] The purpose of step 601 is to obtain the capability inquiry text and, based on this, obtain the actual capability response text corresponding to the target large language model. The capability inquiry text refers to questions designed to test and guide the target large language model's various capability performances, such as directly asking "What can you do?". The actual capability response text is the target model's actual response to these capability-related questions, for example: "I can do all sorts of things! Here are some things I can help you with: 1. Answering questions: I can provide information on a wide range of topics such as science, history, and technology. 2...."

[0117] Step 602: Construct the fifth prompt word field based on the ability inquiry text and the actual ability response text.

[0118] After obtaining the question-and-answer text, a capability constraint field can be constructed in step 602. It can take the form of listing the capability query text and the actual capability response text side-by-side in sequence, for example: "<|im_start|>Use: What can you do?<|im_end|>", "<|im_start|>ChatGPT: I can do all sorts of things! Here are some things I can help you with: 1. Answering questions: I can provide information on a wide range of topics such as science, history, technology, etc...".

[0119] <|im_end|>”

[0120] Step 603: Construct context-aligned prompts based on the first prompt field, the second prompt field, the third prompt field, the fourth prompt field, and the fifth prompt field.

[0121] Finally, in step 603, the fifth cue word field is combined with the previous cue word fields to form a context-aligned cue word that includes the target model's identity and specific capability representation, providing a more comprehensive and detailed constraint on the model. The pre-trained large language model will be given a more accurate and comprehensive behavioral reference, and its semantic understanding and generation capabilities will be fully guided, thereby achieving efficient alignment of the reference target large language model at multiple levels and in multiple scenarios.

[0122] In some embodiments, step 603 may include: generating context-aligned prompts based on a preset layout format and a preset dialogue structure format, according to a first prompt field, a second prompt field, a third prompt field, a fourth prompt field, and a fifth prompt field.

[0123] In embodiments of the present invention, the process of constructing context-aligned cue words is a comprehensive task involving the integration of multiple cue word fields according to a preset layout and dialogue structure format to generate high-quality context-aligned cue words. The format and content of the context-aligned cue words can be precisely constrained and standardized. This helps ensure that the large language model can strictly follow the preset layout and dialogue structure format to generate response text in natural language text form, improving the accuracy and consistency of alignment.

[0124] Step 104: Obtain the target conversation text and add it to the context-aligned cue words so that the pre-trained large language model can process the target conversation text to obtain the conversation response text.

[0125] Step 104 involves adding the acquired target conversation text to the contextual prompts. The target conversation text refers to the natural language text query or instruction entered by the user during dialogue interaction with the large language model. By directly inputting this text into the pre-trained large language model, the model can generate a response equivalent to that of the target large language model. For example, suppose a user asks "What is the meaning of life?" in the input box; this question text can be used as the target conversation text input into the pre-trained large language model. Guided by the contextual alignment prompts constructed in step 103, the pre-trained large language model will generate text following the user's instructions.

[0126] Step 105: Add the conversation response text of the current dialogue round to the context-aligned cue words.

[0127] Step 105 aims to enhance the pre-trained large language model's ability to handle multi-turn conversations by appending the response text generated by the model in the current turn of the conversation to the context-aligned cue words. In this way, the cue words are no longer static but dynamically updated with each new turn of the conversation.

[0128] Taking the above example, suppose the pre-trained large language model generates an explanatory response about "What is the meaning of life?" in this round. This response text will be appended to the original prompt, creating a new context-aligned prompt. When the user continues to ask new related questions, the pre-trained large language model, when generating new responses, can use this updated prompt, which includes the current conversation history, to comprehensively consider the contextual semantics and generate more coherent and reasonable multi-turn responses.

[0129] Understandably, step 105 endows the pre-trained large language model with a certain contextual memory capability, enabling it to continuously accumulate semantic information during the conversation, rather than simply responding to each query independently. This dialogue history accumulation and update mechanism greatly enhances the model's ability to handle multi-turn interactions, making the human-computer dialogue process more natural and smooth, and improving the overall user experience.

[0130] Step 106: In response to obtaining the target conversation text for the next conversation round, input the target conversation text for the next conversation round and the context alignment cue words into the pre-trained large language model to obtain the conversation response text for the next conversation round, and set the next conversation round as the current conversation round.

[0131] In step 106, when the user continues to input new target conversation text, such as asking a new question or giving a new instruction, the next round's conversation text, along with the updated context alignment prompts, is input into the pre-trained large language model. By repeatedly executing steps 105 and 106, the model can continuously accumulate contextual semantic information from multiple rounds of conversation, thus enabling more natural multi-round dialogues. This dialogue history accumulation and update mechanism enhances the model's ability to handle context alignment in multi-round conversations, making the human-computer dialogue process more natural and fluid.

[0132] See Figure 7 In some embodiments, the prompt word construction process of the above model can also be implemented in English. This embodiment selects the aligned Mistral-8*7b pre-trained large language model. The specific type of pre-trained large language model is not limited in this embodiment. The specific steps include the following:

[0133] Step 1: Initialize the Mistral-8*7b pre-trained large language model to enter the text completion task state.

[0134] Step 2: Construct context-aligned text completion task prompts for the Mistral-8*7b pre-trained large language model.

[0135] Step 2.1: Use natural language “Be low is a **correct** conversation between ChatGPT (**GPT4**) and the user, organized in **ChatML** and **Markdown** formats:” to describe the keywords **correct** and formats **ChatML** and **Markdown** in the subsequent text to be completed. The conversation start and end characters are “<|im_start|>” and “<|im_end|>”, respectively, which constitute the first prompt text.

[0136] Step 2.2: Select ChatGPT, a widely mentioned AI assistant name in the pre-training data, as the AI ​​assistant object, i.e., ChatGPT (**GPT4**) from the previous step.

[0137] Step 2.3: Set the system prompt word of the Mi stra l-8*7b pre-trained large language model to the prompt word of the AI ​​assistant ChatGPT itself selected in Step 2.2, "You are ChatGPT, large language model trained by OpenAI, based on the GPT-4 architecture.", to form the second prompt word text.

[0138] Step 2.4: Use natural language such as “You are a he l pful l, respectful l, and honest assi stant.” to describe the usefulness, harmlessness and other safety statements that the large language model intelligent assistant needs to meet, and add them to the system prompt words to form the third prompt word text.

[0139] Step 2.5: Use the text containing self-identity awareness as the first round of dialogue content for the Mistral-8*7b pre-trained large language model, and use it as the fourth cue word text.

[0140] Step 2.5.1: Following the format declared in step 2.1, the user initiates a greeting conversation using the natural language "Hi."

[0141] Step 2.5.2: Following the format declared in step 2.1, fill the actual reply from the smart assistant Chat GPT selected in step 2.2, which includes a self-introduction, “Hey lo! I'm **ChatGPT**. How can I ask you today?” into the response conversation of the Mistral-8*7b pre-trained large language model.

[0142] Step 2.6: Use the text containing self-skill awareness as the second-round dialogue content of the Mistral-8*7b pre-trained large language model to form the fifth cue word text.

[0143] Step 2.6.1: Following the format declared in step 2.1, the user asks the intelligent assistant about its skills using the natural language phrase "What can you do?"

[0144] Step 2.6.2: Following the format declared in Step 2.1, fill the actual response from the intelligent assistant Chat GPT selected in Step 2.2, which includes a self-introduction of its skills, such as “I can do a variety of things! Here are some of the things I can assist you with: 1. Answering questions: I can provide information on a wide range of topics such as science, history, technology, and more.” into the response conversation of the Mistral-8*7b pre-trained large language model.

[0145] Step 2.7: Following the format declared in step 2.1, the user describes the task session instructions using natural language, such as "What is the meaning of life?".

[0146] Step 3: Fill the context of the Mistral-8*7b pre-trained large language model from Step 1 with the prompt words constructed in Step 2, and use the conversation end keyword "<|im_end|>" declared in Step 2.1 as the stopping condition for generation. Utilize the continuation generation capability of the Mistral-8*7 pre-trained large language model; the inferred text generated is the content of the Mistral-8*7b pre-trained large language model's response in this conversation.

[0147] Step 4: Append the text generated in Step 3 to the prompt word constructed in Step 2, and jump to Step 2.7.

[0148] Step 5: Repeat step 4 until the user exits the program, at which point the program terminates.

[0149] Please see Figure 8 This application also provides an alignment device 800 for a large language model, which can implement the above-mentioned large language model alignment method, including:

[0150] The first acquisition module 801 is used to acquire a pre-trained large language model and initialize the pre-trained large language model to enter the text completion task state; wherein, the dataset corresponding to the pre-trained large language model during pre-training is the target dataset;

[0151] The determination module 802 is used to perform model feature retrieval on the target dataset and determine at least one target large language model;

[0152] Module 803 is used to construct context-aligned prompts based on the target large language model and input the context-aligned prompts into the pre-trained large language model so that the pre-trained large language model can align with the target large language model.

[0153] The second acquisition module 804 is used to acquire the target conversation text and add the target conversation text to the context-aligned prompt words so that the pre-trained large language model can perform task processing on the target conversation text to obtain the conversation response text.

[0154] Add module 805 to add the conversation response text of the current dialogue round to the context-aligned prompt.

[0155] The response module 806 is used to respond to the acquisition of the target conversation text of the next conversation round, input the target conversation text of the next conversation round and the context alignment prompt words into the pre-trained large language model, obtain the conversation response text of the next conversation round, and set the next conversation round as the current conversation round.

[0156] This application proposes a method, electronic device, and storage medium for aligning a large language model. The method includes acquiring a pre-trained large language model and initializing it to enter a text completion task state. The dataset corresponding to the pre-trained large language model during pre-training is the target dataset. Model feature retrieval is performed on the target dataset to determine at least one target large language model. Context alignment prompts are constructed based on the target large language model and input into the pre-trained large language model so that the pre-trained large language model aligns with the target large language model. Target conversation text is acquired and added to the context alignment prompts so that the pre-trained large language model processes the target conversation text to obtain conversation response text. The following steps are executed iteratively: adding the conversation response text of the current conversation round to the context alignment prompts; in response to acquiring the target conversation text of the next conversation round, inputting the target conversation text of the next conversation round and the context alignment prompts into the pre-trained large language model to obtain the conversation response text of the next conversation round, and setting the next conversation round as the current conversation round.

[0157] In contrast to the problems in the background art, the large language model alignment method provided in this application utilizes prompting engineering techniques. It constructs context alignment prompts by referencing the characteristics of the target large language model and leveraging the text continuation reasoning capabilities of a pre-trained large language model. These prompts are then input into the original pre-trained large language model, effectively guiding and aligning its behavior. This enables the pre-trained large language model, which originally only possessed text completion capabilities, to exhibit advanced capabilities similar to the target large language model, such as instruction following and task processing. Compared to traditional model fine-tuning training methods, this method avoids the high human and computational costs of constructing new instruction following datasets and fine-tuning training models. It requires no additional dedicated manual data or fine-tuning training, reducing alignment costs, improving alignment performance, and achieving rapid alignment compared to existing large language model alignment methods.

[0158] This application also provides an electronic device, including:

[0159] At least one memory;

[0160] At least one processor;

[0161] At least one program;

[0162] The program is stored in memory, and the processor executes the at least one program to implement the large language model alignment method described above in this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), in-vehicle computers, etc.

[0163] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0164] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0165] The memory 902 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the large language model alignment method of the embodiments of this application.

[0166] The input / output interface 903 is used to implement information input and output;

[0167] The communication interface 904 is used to enable communication between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0168] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0169] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0170] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described large language model alignment method.

[0171] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0172] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0173] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0175] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0176] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0177] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0179] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0181] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for aligning large language models, characterized in that, The method comprises: obtaining a pre-trained large language model and initializing the pre-trained large language model into a text completion task state; wherein the data set corresponding to the pre-trained large language model during pre-training is a target data set; performing model feature retrieval on the target data set to determine at least one target large language model; constructing a context alignment prompt word according to the target large language model, and inputting the context alignment prompt word into the pre-trained large language model, so that the pre-trained large language model refers to the target large language model for alignment; obtaining target conversation text, adding the target conversation text to the context alignment prompt word, so that the pre-trained large language model processes the target conversation text to obtain conversation response text; recursively performing the following steps: adding the conversation response text of the current dialogue round to the context alignment prompt word; in response to obtaining target conversation text of the next dialogue round, inputting the target conversation text of the next dialogue round and the context alignment prompt word into the pre-trained large language model to obtain the conversation response text of the next dialogue round, and setting the next dialogue round as the current dialogue round.

2. The method of Claim 1, wherein, The context alignment prompt word comprises a first prompt word field, and the context alignment prompt word is constructed according to the target large language model, comprising: obtaining a target keyword, a preset layout format, and a preset dialogue structure format; generating the format constraint field according to the preset layout format and the preset dialogue structure format; generating a first prompt word field according to the target name of the target large language model, the target keyword, the layout format, and the dialogue structure format; constructing the context alignment prompt word according to the first prompt word field.

3. The method of Claim 2, wherein, The context alignment prompt word further comprises a second prompt word field, and the context alignment prompt word is constructed according to the target large language model, comprising: obtaining a target name and a target system word of the target large language model; constructing the second prompt word field according to the target name and the target system word; constructing the context alignment prompt word according to the first prompt word field and the second prompt word field.

4. The alignment method of a large language model according to claim 3, characterized in that, The context alignment prompt word further comprises a third prompt word field, and the context alignment prompt word is constructed according to the target large language model, comprising: obtaining a first target security word of the target large language model; wherein the first target security word is used to guide the large language model to generate a security text; generating the third prompt word field based on the first target security word; constructing the context alignment prompt word according to the first prompt word field, the second prompt word field, and the third prompt word field.

5. The method of Claim 4, wherein, The context alignment prompt word further comprises a fourth prompt word field, and the context alignment prompt word is constructed according to the target large language model, comprising: obtaining an identity inquiry text, and obtaining an actual identity reply text of the target large language model for the identity inquiry text based on the identity inquiry text; construct the fourth prompt field according to the identity query text and the actual identity reply text; construct the context alignment prompt according to the first prompt field, the second prompt field, the third prompt field, and the fourth prompt field.

6. The alignment method of a large language model according to claim 5, wherein, The context alignment prompt further includes a fifth prompt field, and the constructing the context alignment prompt according to the target large language model includes: obtaining an ability query text, and obtaining an actual ability reply text of the target large language model based on the ability query text; constructing the fifth prompt field according to the ability query text and the actual ability reply text; constructing the context alignment prompt according to the first prompt field, the second prompt field, the third prompt field, the fourth prompt field, and the fifth prompt field.

7. The alignment method of a large language model according to claim 6, characterized in that, The constructing the context alignment prompt according to the first prompt field, the second prompt field, the third prompt field, the fourth prompt field, and the fifth prompt field includes: generating the context alignment prompt according to the first prompt field, the second prompt field, the third prompt field, the fourth prompt field, and the fifth prompt field based on the preset layout format and the preset dialogue structure format.

8. An aligning apparatus of a large language model, characterized by, The device includes: The first obtaining module is configured to obtain a pre-trained large language model and initialize the pre-trained large language model into a text completion task state; wherein the pre-trained large language model corresponds to a target data set during pre-training; The determining module is configured to perform model feature retrieval on the target data set to determine at least one target large language model; The constructing module is configured to construct a context alignment prompt according to the target large language model, and input the context alignment prompt into the pre-trained large language model, so that the pre-trained large language model aligns with the target large language model; The second obtaining module is configured to obtain a target dialogue text, add the target dialogue text to the context alignment prompt, so that the pre-trained large language model processes the target dialogue text to obtain a dialogue response text; The adding module is configured to add the dialogue response text of the current dialogue round to the context alignment prompt; The response module is configured to, in response to obtaining a target dialogue text of a next dialogue round, input the target dialogue text of the next dialogue round and the context alignment prompt into the pre-trained large language model to obtain the dialogue response text of the next dialogue round, and set the next dialogue round as the current dialogue round.

9. An electronic device, comprising: The computer program is executed by the processor to implement the alignment method of the large language model according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the alignment method of the large language model according to any one of claims 1 to 7.

Images