Model training method, model reasoning method and computing device cluster

Abstract

The embodiment of the invention provides a model training method, a model reasoning method and a computing device cluster. The invention further provides a computer readable storage medium. The invention relates to the technical field of computers, and provides a model training method which is applied to computing equipment and comprises the steps that a first strategy instruction is generated according to original input, the original input comprises a user question, the first strategy instruction is related to the original input, and the first strategy instruction is used for guiding generation of a first reply; the first reply is a reply for the original input; the original input and the first strategy instruction are input into the model, a first reply output by the model is obtained, and the first reply follows the first strategy instruction; the quality of the first reply is checked, a first check result is generated, and the first check result comprises that the first reply passes the check or does not pass the check; and under the condition that the first reply passes the check, performing supervised fine tuning on the model according to the original input, the first strategy instruction and the first reply. According to the model training method provided by the embodiment of the invention, the quality of the reply generated by the model can be improved, and possible errors in the reply generation process can be avoided.

Description

A model training method, a model inference method, and a computing device cluster Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, a model inference method, and a computing device cluster. Background Technology

[0002] With the development of artificial intelligence technology, the application fields of artificial intelligence models are becoming more and more extensive. For example, large language models (LLMs) are used to solve natural language generation tasks.

[0003] In the application scenarios of artificial intelligence models, model performance includes model safety capabilities, model value level, and model complex reasoning ability. Model performance directly affects the final model service effect.

[0004] Therefore, a new model training method is needed to improve model performance. Summary of the Invention

[0005] To address the issue of how to improve model performance, this application provides a model training method, a model inference method, and a computing device cluster. This application also provides a computer-readable storage medium.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] Firstly, this application provides a model training method, the method comprising:

[0008] A first strategy instruction is generated based on the original input, wherein the original input includes a user question, the first strategy instruction is related to the original input, and the first strategy instruction is used to guide the generation of a first response, which is a response to the original input.

[0009] The original input and the first policy instruction are input into the model, and the first response output by the model is obtained, wherein the first response conforms to the first policy instruction; in one implementation, the model is a pre-trained model after pre-training.

[0010] Check the quality of the first response and generate a first check result, which includes whether the first response passes the check or fails the check;

[0011] If the first response passes the check, the model is then fine-tuned under supervision based on the original input, the first policy instruction, and the first response.

[0012] According to the model training method in the first aspect, the ability to generate responses based on policy instructions is injected into the model during the model training phase. This method can improve the quality of the model-generated responses and avoid potential errors during response generation.

[0013] Ideally, based on policy instructions, the model can generate higher-quality responses; that is, responses generated based on policy instructions should pass the inspection. However, in real-world scenarios, policy instructions do not necessarily improve response quality. Possible reasons include overly broad policy instructions or incorrect guidance. This can lead to responses generated based on policy instructions failing the inspection.

[0014] In one implementation of the first aspect, the method further includes:

[0015] If the first response fails the check, an error analysis report is generated for the first response;

[0016] Based on the error analysis report, a second strategy instruction is generated according to the original input. The second strategy instruction is related to the original input and is used to guide the generation of a second response, which is a response to the original input.

[0017] The original input and the second policy instruction are input into the model to obtain the second response output by the model, wherein the second response conforms to the second policy instruction;

[0018] Check the quality of the second response and generate a second check result, wherein the second check result includes whether the second response passes the check or fails the check;

[0019] When the second response passes the check, supervised fine-tuning is performed based on the original input, the second strategy instruction, and the second response.

[0020] According to the method of this application embodiment, when the response fails the check, the policy instruction is refreshed, so that the policy instruction finally used for training can effectively guide the generation of the response, thereby improving the model training effect and model training efficiency.

[0021] The model training method described above refreshes policy instructions based on error analysis reports of responses that fail the checks, making the policy instructions more compatible with model training and learning. This allows the policy instructions to be effectively used to train the model, improving both the model training effect and efficiency.

[0022] In one implementation of the first aspect, a strategy instruction is constructed based on the original input, inspired by the expectation-maximization algorithm, to guide the generation of the final response.

[0023] Specifically, in the use cases of the model, task execution needs to follow certain basic principles. Therefore, in one implementation of the first aspect, generating a first policy instruction based on the original input includes:

[0024] Based on the fundamental principles that the model should follow when performing tasks, a first policy instruction is constructed for the original input.

[0025] Furthermore, in the application scenarios of the model, the basic principles that the model needs to follow when performing tasks differ depending on the task. Although it is impossible to completely predict all the tasks that the model may perform, the task type of each example is known during the construction of the training data. Therefore, for a specific task type, the basic principles that should be followed when performing tasks of that type can be obtained.

[0026] Therefore, in one implementation of the first aspect, based on the fundamental principles that the model should follow when performing tasks, a first policy instruction is constructed for the original input, including:

[0027] Based on the basic principles that the model should follow when performing the first type of task, policy instructions are constructed for the original input, where the task type of the first type of task is the task type corresponding to the original input.

[0028] Based on the model training method described above, policy instructions are constructed for the original input. These policy instructions are tailored to each instance / problem, enabling them to capture the dimensions required for each instance / problem, thus allowing them to better guide response generation.

[0029] In one implementation of the first aspect, the first strategy instruction is a natural language instruction used to improve the quality of the first response and avoid errors that may occur during the generation of the first response.

[0030] In one implementation of the first aspect, supervised fine-tuning is performed based on the original input, the first policy instruction, and the first response, including:

[0031] The original input, the first policy instruction, and the first response are used as fine-tuning training data.

[0032] Supervised fine-tuning of the model is performed based on the fine-tuning training data.

[0033] In one implementation of the first aspect, the original input also includes search information, wherein the search information is information obtained by searching based on the user's question.

[0034] In one implementation of the first aspect, supervised fine-tuning is performed based on the original input, the first policy instruction, and the first response, including:

[0035] By combining the user question in the original input with the search information, atomic capability tasks are extracted from the search information.

[0036] Based on the search information, the atomic ability task responses for the corresponding atomic ability tasks are summarized according to the atomic ability tasks.

[0037] The model is then fine-tuned under supervision based on the atomic capability tasks and their responses.

[0038] In one implementation of the first aspect, supervised fine-tuning is performed based on the original input, the first policy instruction, and the first response, including:

[0039] By combining the user question in the original input with the search information, atomic capability tasks are extracted from the search information.

[0040] Obtain the atomic strategy instructions for the corresponding atomic capability task;

[0041] Based on the search information, obtain the atomic capability task response for the corresponding atomic capability task according to the atomic capability task and atomic policy instruction;

[0042] The model is fine-tuned under supervision based on atomic capability tasks, atomic policy instructions, and atomic capability task responses.

[0043] In one implementation of the first aspect, obtaining the atomic policy instruction for the corresponding atomic capability task includes:

[0044] Simplify the first policy instruction based on atomic capability tasks to obtain atomic policy instructions.

[0045] The model training method described above uses atomic instruction compliance training data to form the original input and policy instructions corresponding to the checked responses. Training the model using this atomic instruction compliance training data can improve the model's ability to comply with policy instructions, enhance the quality of the responses generated by the model, and more effectively avoid errors that may occur during the response generation process.

[0046] Secondly, this application provides a model reasoning method, the method comprising:

[0047] Generate the first strategy instruction based on the original input;

[0048] The original input and the first strategy instruction are input into the first model, and the first response output by the first model is obtained, wherein the first response conforms to the first strategy instruction.

[0049] Based on the model reasoning method in the second aspect, introducing policy instructions during the reasoning process to guide the generation of responses can improve the quality of the responses generated by the model and avoid errors that may occur during the response generation process.

[0050] In one implementation of the second aspect, generating a first policy instruction based on the original input includes:

[0051] The original input is fed into the second model, and the first policy instruction output by the second model is obtained.

[0052] In one implementation of the second aspect, generating a first policy instruction based on the original input includes:

[0053] Input the original input into the first model and obtain the first policy instruction output by the first model.

[0054] Furthermore, in another implementation of the second aspect, policy instructions are not generated based on the original input during inference. Instead, generic policy instructions are pre-generated before inference. During inference, the pre-generated generic policy instructions are invoked, and the original input and the invoked policy instructions are input into the model to obtain the model's output response conforming to the policy instructions.

[0055] Alternatively, multiple (or sets of) policy instructions can be pre-generated before inference. During inference, policy instructions (or sets of policy instructions) that match the original input are called from the pre-generated policy instructions. The original input and the called policy instructions are then input into the model, and the model outputs a response that conforms to the policy instructions.

[0056] Thirdly, this application provides a computing device cluster, the computing device cluster including at least one computing device, each computing device including a memory and a processor;

[0057] The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in the first aspect or the second aspect.

[0058] Fourthly, this application provides a computer program product containing instructions that, when run by a computing device system, cause a computing device cluster to perform the method described in the first or second aspect of claim.

[0059] Fifthly, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a computer system, perform the method described in the first or second aspect. Attached Figure Description

[0060] Figure 1 shows a schematic diagram illustrating the model benefits brought about by the task prompting engineering;

[0061] Figure 2 is a schematic diagram of a model training device according to an embodiment of this application;

[0062] Figure 3 is a flowchart of a model training method according to an embodiment of this application;

[0063] Figure 4 is a schematic diagram of a model training device according to an embodiment of this application;

[0064] Figure 5 is a flowchart of a model training method according to an embodiment of this application;

[0065] Figure 6 shows a schematic diagram of model training data input according to an embodiment of this application;

[0066] Figure 7 is a schematic diagram of the structure of a reasoning device according to an embodiment of this application;

[0067] Figure 8 is a flowchart of a reasoning method according to an embodiment of this application;

[0068] Figure 9 shows a comparison chart of model inference response quality according to an embodiment of this application;

[0069] Figure 10 is a schematic diagram of a computing device structure according to an embodiment of this application;

[0070] Figure 11 is a schematic diagram of a computing device cluster according to an embodiment of this application;

[0071] Figure 12 is a schematic diagram of a computing device cluster network connection according to an embodiment of this application;

[0072] Figure 13 is a schematic diagram of a system architecture according to an embodiment of this application. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0074] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0075] The working process of a large model typically includes a training phase (process) and an inference phase (process).

[0076] During the training phase, the large model uses deep learning technology and multi-layer neural networks to learn and optimize the massive amounts of input data, adjusting the model's parameters to enable it to make accurate predictions on the input data.

[0077] Specifically, the training process of a large language model (LLM) includes a pre-training process and a supervised fine-tuning (SFT) process.

[0078] The goal of pre-training is to generate a pre-trained model by training the LLM model on a relevant but large task, enabling the LLM model to learn general feature representations.

[0079] SFT is used to fine-tune a pre-trained model by using labeled downstream task data.

[0080] The SFT process typically uses a large labeled dataset (X->Y) to fine-tune all or some parameters of the pre-trained model. Therefore, supervised fine-tuning is often called instruction fine-tuning, where X represents the original input instruction fed to the model and Y represents the response text that the model is expected to fit.

[0081] For example, an X->Y training example is as follows:

[0082] X: How should I respond when someone comments on my weight?

[0083] Y: Hello, sometimes it can be uncomfortable and confusing to hear others comment on your weight. Here are a few ways to understand and respond to comments about your weight:

[0084] 1. **Unintentional Remarks:** Sometimes, others may only make unintentional remarks without any real malice. In this case, you can respond humorously, such as, "Yeah, I'm the most composed one among us!" to ease the tension.

[0085] 2. **With a Joke:** If the other person is joking, you can respond with a joke to keep the atmosphere light. For example, you could say, "You know me too well; you'll have to work harder to catch up." This will lighten the mood for everyone.

[0086] 3. **Positive Response:** Expressing your feelings directly and sincerely is also a method. For example, you could say, "That made me a little uncomfortable, and I hope you can be mindful of it." This not only reflects your feelings but also lets the other person know your bottom line.

[0087] 4. **Confident Response:** If you feel confident, you can also look at this issue from the perspective of health and strength. For example, "Actually, I feel very healthy and stronger than many people."

[0088] Some people gain weight due to long-term exercise and training, which isn't a bad thing; from a health perspective, it's a sign of strength and endurance. Hopefully, these suggestions will help you better handle similar comments and feel more confident about yourself!

[0089] Once the large model has been trained, it can be used in the inference phase. The inference phase refers to applying the trained model to new, unseen data, building upon the completed model training.

[0090] The inference phase is where the large model is applied to real-world service scenarios. The inference phase showcases the performance of the trained model; for example, model performance can include security capabilities, value levels, and complex reasoning abilities.

[0091] For challenging natural language generation tasks (such as answering value-based questions, answering reasoning questions, and generating long texts), a feasible technical solution to improve model performance is to increase the amount of instruction fine-tuning data. This involves introducing more labeled data to enhance the model's coverage of the input distribution. In other words, this means increasing the training workload of the SFT process.

[0092] For example, Table 1 below compares the version iterations of a certain model.

[0093] Table 1

[0094]

[0095] As shown in Table 1, in its second version, the SFT process used a total of one million (1M) human-annotated data points, while in the third version, the amount of data for fine-tuning the SFT process increased tenfold, using a total of 10M+ human-annotated data points.

[0096] While increasing the amount of fine-tuning data can lead to better model performance, collecting fine-tuning data is time-consuming and labor-intensive. More importantly, training pre-trained models directly on this X->Y label data often results in less than ideal learning outcomes. This is especially true for pre-trained models with relatively weak foundational capabilities; directly fitting the same X->Y label data fails to establish the underlying causal relationships, leading to poor training results and poor generalization.

[0097] Another feasible technical solution is to improve model performance through prompt engineering. Prompt engineering refers to guiding a large model to generate high-quality articles on specific topics, answer specific questions, or engage in meaningful dialogues by designing appropriate prompts during inference.

[0098] In prompting engineering, the quality and accuracy of prompt words are crucial for generating expected results. In other words, a key issue in prompting engineering is how to construct a good prompt for a user's original input command X.

[0099] Figure 1 shows a schematic diagram illustrating the model benefits brought about by the task prompting project.

[0100] As shown in Figure 1, the vertical axis represents the model benefit. The four horizontal bars represent no prompts and three different prompts, respectively. By comparing the three bars with prompts and the bar without prompts, optimizing the task description and instruction hints in the input prompts can significantly improve the model benefit.

[0101] In the implementation of prompting engineering, specialized prompt words are developed for different tasks to improve the model's response performance in the corresponding tasks. However, prompting engineering is often built for a certain category of tasks, which is not applicable to real-world open application scenarios. Furthermore, the generated instructions lack interpretability and are insufficient to model the intermediate latent variables of complex task questions and answers, raising questions about generalization.

[0102] For example, in one technical solution, an Automatic Prompt Engineer (APE) is used to optimize prompt words.

[0103] AutoSuggestion Engineer (APE) is an automated instruction generation and selection framework designed to improve the performance of large language models when performing specific tasks. The core idea of ​​APE is to treat instructions as "programs" and guide large models to achieve better performance by searching and optimizing these "programs."

[0104] APE's focus includes: given a large model, how to perform prompting engineering to generate better instructions to guide the large model in responding.

[0105] Specifically, in one embodiment, the method for APE to find the optimal prompt mainly includes the following four stages.

[0106] 1. Instruction Candidate Generation: APE uses a large model to generate a set of instruction candidates. These candidates are generated by guiding the large model through a small set of input-output examples; this process can be seen as an initial proposal of instructions.

[0107] 2. Evaluating Instruction Quality: To evaluate the quality of instruction candidates, APE defines scoring functions such as execution accuracy and log probability. These functions measure the consistency between the model-generated data and the dataset.

[0108] 3. Iterative Monte Carlo Search: APE treats the instruction generation process as a black-box optimization problem, using a large model to guide the search process. It employs heuristics to search the candidate instruction space, seeking instructions that maximize the scoring function. APE uses an iterative Monte Carlo search method to improve the instruction candidates. In each round of the search, it evaluates the current best candidates, generates new variants based on these candidates, and then selects a new candidate set for the next round of searching.

[0109] 4. Instruction Selection and Evaluation: After multiple rounds of search, APE selects the instruction with the highest score as the final instruction. This instruction is then used to guide the large model in performing the task.

[0110] From APE's perspective, the instructions used for guidance are specific to a class of tasks, and each task has only one final instruction that does not change with different instances or problems. In real-world scenarios, different problems will have different optimal instructions, and a static final instruction cannot cover all aspects of every problem.

[0111] Specifically, APE has the following drawbacks:

[0112] (1) Only applicable to the reasoning stage: As a type of prompting engineering, APE is only applicable to the reasoning process. Since it has not been trained, the large model's own instruction generation and related capabilities have not been improved accordingly.

[0113] (2) Strong dependence on the large model itself: The performance of APE depends heavily on the underlying capabilities of the large model used. If the model itself has biases or limitations, the instructions generated by APE may not achieve optimal results. Therefore, APE is applicable to large-scale models (e.g., InstructGPT of 175B) but not to small-scale models.

[0114] (3) Static final instruction: After the search is completed, APE has only one optimal instruction for each task, such as question answering, and it will not change with different questions within the task. In real-world scenarios, different questions will have different optimal instructions, and a static final instruction cannot cover all aspects of each question.

[0115] (4) Generalization ability: Although APE performs well in specific tasks (complex reasoning), its generalization ability to other tasks or domains (such as long text generation) is limited.

[0116] Furthermore, in another feasible technical solution, for highly challenging natural language generation tasks, particularly citation-based summary generation tasks, blueprints are used to implement SFT, thereby optimizing model training effects and improving model performance.

[0117] The task of generating summaries with citations involves extracting information from a set of documents most relevant to the user's question and generating a comprehensive summary while ensuring that each statement in the summary has a corresponding source citation as supporting evidence.

[0118] A blueprint is a set of questions that need to be answered in the generated summary.

[0119] Specifically, in one embodiment, blueprints include two types: abstract blueprints and extractable blueprints. Abstract blueprints generate the problem from scratch based on the user's question and related documentation, without directly copying the question from the input text. Extractable blueprints, on the other hand, copy the question from related documentation to ensure the accuracy of the reference.

[0120] While using blueprints to assist in answer generation can improve the accuracy and coverage of information in the answers, the blueprint approach still has the following problems:

[0121] (1) Limitations of blueprints: As a set of problems, blueprints can only be used to assist large models in finding information related to blueprints.

[0122] (2) Limitations of model type: The implementation of the blueprint scheme mainly relies on the fine-tuning encoder-decoder model (LongT5), which cannot be well adapted to the mainstream decoder-only models (such as GPT, LLaMA, etc.).

[0123] (3) Generalization ability of the model: Although the blueprint scheme performs well on summary datasets, for complex tasks (e.g., long text question answering), the blueprint scheme performs worse than large models using hint engineering on large datasets.

[0124] (4) Dataset dependency: The blueprint generation for model learning depends on specific datasets (such as AQuAMuSe and ALCE). If the generated blueprint does not help the model better understand the user's question and related documents, then the intermediate blueprint is meaningless.

[0125] To address the issue of how to improve model inference performance and enhance model performance, one embodiment of this application provides a self-suggestion method for training perception models. The model adaptively responds with prompts for each user's original instruction input and follows its own prompts, ultimately improving the response quality for high-difficulty tasks.

[0126] In the model training method provided in this application embodiment, the quality of the model's response is improved based on policy instructions. Policy instructions are related to the original input to the model and are used to guide the model in generating responses to the original input. Policy instructions include requirements for generating responses in multiple dimensions, such as response outline, organizational form, and writing style. Based on policy instructions, the quality of the responses generated by the model can be improved, and errors that may occur during the response generation process can be avoided.

[0127] Specifically, the method in this application introduces policy instructions during the training and inference phases to enable model self-prompting and improve end-to-end response quality.

[0128] One embodiment of this application provides a model training method and a model training apparatus for implementing the model training method.

[0129] In the model training method of this application embodiment, during the training process, given X, the model not only fits the expected Y, but also fits Z at the same time, so as to improve the learning effect and learning efficiency of large model instruction fine-tuning.

[0130] Specifically, in one embodiment, the user's original input (e.g., a math problem to be solved) fed to the model during the model training phase is labeled X, and the final expected response (e.g., the final answer to the math problem) is labeled Y. Z is a series of natural language instructions related to X, which can be used to prompt improvements in the quality of Y and avoid errors that may occur during the generation of Y.

[0131] The method described in this application can be applied to a single electronic device (or computing device) or a system (or cluster of computing devices) consisting of multiple electronic devices.

[0132] Figure 2 is a schematic diagram of a model training device according to an embodiment of this application.

[0133] As shown in Figure 2, the model training device 200 includes:

[0134] Strategy instruction generation module 201 is used to generate Z based on X.

[0135] The inference module 202 is used to input X and Z into the model and obtain the Y that conforms to Z from the model output.

[0136] The inspection module 203 is used to inspect the quality of Y obtained by the inference module 202 and generate inspection results, including whether Y passes the inspection or fails the inspection. Furthermore, when Y fails the inspection, the inspection module 203 also generates an error analysis report E.

[0137] The strategy instruction generation module 201 is also used to regenerate Z based on X, according to the error analysis report E generated by the inspection module 203.

[0138] The training module 204 is used to take the Y that passed the check and the corresponding X and Z as fine-tuning training data when Y passes the check, and to perform SFT on the model based on the fine-tuning training data.

[0139] Figure 3 shows a flowchart of a model training method according to an embodiment of this application.

[0140] The model training device 200 shown in Figure 2 executes the following process as shown in Figure 3 to achieve model training.

[0141] S300, the policy instruction generation module 201 generates a policy instruction (e.g., policy instruction Z) based on the original input (e.g., the original input is X). With the original input as X and the policy instruction as Z, the process of S300 is represented as X->Z.

[0142] Specifically, in one embodiment, the original input includes at least a user question. For example, one original input might be "How many days of rain were there last week?"

[0143] In another embodiment, the original input includes a user question and search information obtained by searching based on the user question.

[0144] In other words, in the use case of the model, after obtaining the user's question, it is necessary to perform a search based on the user's question, and input the search information and the user's question into the model so that the model can generate a response.

[0145] For example, a given original input contains:

[0146] User question: "How many days of rain were there last week?"

[0147] In addition, it retrieves last week's weather information based on user questions.

[0148] Specifically, in one embodiment, inspired by the Expectation-maximization algorithm (EM), policy instructions are constructed based on the original input to guide the generation of the final response.

[0149] Specifically, in the application scenarios of the model, task execution needs to follow certain basic principles. For example: the model's output response must be accurate, and irrelevant content cannot be output, nor can incorrect conclusions be output as correct conclusions; the model's output response must be readable, and content that cannot be understood / recognized cannot be output.

[0150] In one embodiment, in S300, strategy instructions are constructed for the original input based on the basic principles that the model should follow when performing tasks.

[0151] Furthermore, in one implementation, strategy instructions are constructed using manual annotation based on the fundamental principles that should be followed during task execution.

[0152] In another implementation, strategy instructions are constructed using automatic model annotation based on the fundamental principles that should be followed during task execution.

[0153] Furthermore, in the application scenarios of the model, the basic principles that the model needs to follow when performing tasks differ depending on the task. Although it is impossible to completely predict all the tasks that the model may perform, the task type of each example is known during the construction of the training data. Therefore, for a specific task type, the basic principles that should be followed when performing tasks of that type can be obtained.

[0154] For example, for a task that answers user questions directly through text, the basic principles for performing this task should include at least the following:

[0155] The answer must be accurate and cannot contain any content whose correctness is questionable. It must not use descriptive methods that may cause ambiguity. If you cannot answer accurately, you must clearly state this.

[0156] The values ​​expressed in the answers must be positive, and must not contain negative or even incorrect values.

[0157] The text of the answer must be readable, and the language structure must not be illogical.

[0158] In one embodiment, in S300, a strategy instruction Z is constructed for the original input X based on the basic principles to be followed when executing a type B task, wherein the task type of the type B task is the task type corresponding to the original input X.

[0159] For example, a basic guideline for generating responses based on internet search results is as follows:

[0160] Focus on the following task: responding to user requests based on internet reference information.

[0161] Here are some dimensions to consider when receiving a response:

[0162] I. Intent Understanding

[0163] The response should align with the user's intent and needs. Specifically:

[0164] (1) For requests with ambiguous or unclear intentions, they can be classified and discussed according to the circumstances;

[0165] (2) For requests with seriously unclear intentions, one should refuse to answer and clarify rather than give a hasty reply;

[0166] (3) It can identify and respond reasonably to requests that are deliberately set up as factual or other traps.

[0167] II. Logical Framework

[0168] The response should have a clear and appropriate logical framework, organize information in a reasonable and orderly manner, and fully ensure readability.

[0169] For example, presenting the conclusion first, followed by explanation; listing points; discussing by category; developing the information chronologically; or developing the information step by step. This dimension should be given special consideration, especially when there are conflicting or contradictory references.

[0170] III. Smoothness

[0171] The language is fluent and coherent, with no repetition of information or self-contradiction.

[0172] IV. Information Content

[0173] The response content should satisfy the user's request to the greatest extent possible, providing necessary and comprehensive information and knowledge to answer the user's questions, without omitting key information, and without introducing irrelevant information.

[0174] V. Accuracy of Facts and Common Sense

[0175] (1) When the reference information is relevant and there are no obvious factual or common-sense errors, the response should be strictly faithful to the reference information in its factual description. When the reference information cannot provide a direct answer but can provide necessary material for reasoning, accurate or reasonable reasoning should be made.

[0176] (2) When the reference information cannot help with the request for a reply, such as if it is irrelevant or contains obvious common sense and factual errors, it should be rejected and supplemented with existing internal knowledge to ensure that the facts and common sense are consistent with mainstream understanding.

[0177] VI. Timeliness

[0178] (1) It can identify the timeliness requirements of user requests and select the matching part from the reference information to answer.

[0179] (2) For some time-sensitive key information, the corresponding time point should be added when describing the information to help users understand it better.

[0180] VII. Safety and Values

[0181] (1) The response content complies with general security and values ​​requirements and does not cite or implement sensitive, unsafe, harmful or controversial information.

[0182] (2) In particular, when a user requests or references information that involves a specific company, its leaders and products, the response should not damage the company’s reputation and should focus on presenting positive and encouraging information.

[0183] 8. Empathy

[0184] For user requests that have emotional needs, the response should provide emotional value rather than just focusing on providing objective knowledge, thereby creating an emotional resonance with the user.

[0185] IX. Traceability Requirements

[0186] If a segment from the reference information is used, it should be cited in the corresponding position in the reply using the [serial number].

[0187] Based on the above basic principles, the following policy instruction examples can be constructed as shown in Table 2.

[0188] Table 2

[0189]

[0190] According to the model training method of this application embodiment, a strategy instruction is constructed for the original input. The strategy instruction is tailored for each instance / problem, so that the strategy instruction can capture the dimensions required for each instance / problem, thereby enabling the strategy instruction to better guide the response generation.

[0191] In S301, the inference module 202 inputs the original input and policy instructions into the model and obtains the model's output response that conforms to the policy instructions. Specifically, with the original input as X, the policy instruction as Z, and the response as Y, the process in S301 can be represented as: X, Z->Y.

[0192] Specifically, in one embodiment, the model is a pre-trained model obtained through pre-training.

[0193] Specifically, in one embodiment, the user's original input is concatenated with policy instructions and input to the model. The model generates a response that conforms to the policy instructions based on the policy instruction labeled data.

[0194] Ideally, based on policy instructions, the model can generate higher-quality responses; that is, responses generated based on policy instructions should pass the inspection. However, in real-world scenarios, policy instructions do not necessarily improve response quality. Possible reasons include overly broad policy instructions or incorrect guidance. This can lead to responses generated based on policy instructions failing the inspection.

[0195] Therefore, after S301, S302 is executed in response to the reply obtained in S301.

[0196] S302, the inspection module 203 checks the quality of the response obtained by the reasoning module 202 and generates an inspection result, wherein the inspection result includes whether the response passes the inspection or fails the inspection.

[0197] If a reply fails the check, it is considered invalid.

[0198] If the check result in S302 is a failure, then S304 is executed.

[0199] S304, Inspection module 203 generates an error analysis report for responses that fail the inspection.

[0200] Specifically, in one embodiment, error report analysis can achieve automated evaluation based on evaluation templates and a large language model, assessing the quality of responses from multiple dimensions and obtaining a comprehensive score. An example of an evaluation template input into a large language model is as follows:

[0201] A user question and answer will be provided.

[0202] The task is to evaluate the quality of the responses from the multiple dimensions provided below.

[0203] Current time

[0204] $curr_time

[0205] User Issues

[0206] $query

[0207] Response to be evaluated

[0208] $response

[0209] Evaluation method:

[0210] According to the evaluation criteria mentioned below for each dimension, if there are no errors, no analysis content will be output for that dimension; if it cannot be evaluated, the corresponding error count will be 0; if there are errors, a specific error analysis will be given, and the error count for the corresponding dimension will be incremented by 1.

[0211] Finally, a comprehensive score will be given for each response to the evaluation, ranging from a minimum of 1 point to a maximum of 5 points. An overall analysis of no more than 100 words will also be provided.

[0212] Evaluation criteria for each dimension:

[0213] 1. Correlation:

[0214] It is only necessary to determine whether the answer has made an effort to understand the user's intent in the question.

[0215] If the user's intent is seriously unclear, difficult to understand, or lacks necessary information, the guiding remarks for the response should first point out that the user's question is unclear or not specific, and then discuss it in categories.

[0216] 2. Structural:

[0217] Judge the readability of the answer and whether the information is presented well.

[0218] If multiple viewpoints need to be discussed, examples provided, or answers given step-by-step, then discuss them in separate lines or paragraphs or in Markdown format, and list them according to the required hierarchy.

[0219] 3. Smoothness:

[0220] Judge whether the language of the answer is fluent and natural, and there should be no contradictions, semantic repetitions, or repeated mentions of the same content;

[0221] And when discussing segmented columns, we need to ensure that the logical structure of the columns is clear and that there are no duplicate or mutually inclusive columns.

[0222] 4. Richness:

[0223] Judge whether the answer provides a rich and usable response, whether it is well-founded, and whether it does not omit key information.

[0224] Please answer in the following JSON format:

[0225] {

[0226] "Correlation":{"Analysis":[],"Error Count":?"},

[0227] "Structural":{"Analysis":[],"Error Count":?"},

[0228] "Fluency":{"Analysis":[],"Number of Errors":?"},

[0229] "Richness":{"Analysis":[],"Number of Errors":?"},

[0230] "Comprehensive":{"Analysis":[],"Score":?"},

[0231] }

[0232] S305, the policy instruction generation module 201 generates new policy instructions based on the error analysis report and the original input.

[0233] Specifically, in S305, the implementation of generating new policy instructions based on the original input can refer to S300. That is, in S305, under the constraints of the basic principles, new policy instructions are constructed based on the original input with reference to the error analysis report.

[0234] Because error analysis reports (model error analysis) are introduced in the construction of new policy instructions, the final new policy instructions will be more closely matched to the model training and learning.

[0235] According to the method of this application embodiment, when the response fails the check, the policy instruction is refreshed, so that the policy instruction finally used for training can effectively guide the generation of the response, thereby improving the model training effect and model training efficiency.

[0236] According to the method of this application embodiment, the policy instructions are refreshed based on the error analysis report of the response that fails the check, so that the policy instructions are more suitable for model training and learning, thereby enabling the policy instructions to be effectively used to train the model, improving the model training effect and model training efficiency.

[0237] Specifically, in one embodiment, in S305, the policy instruction is automatically refreshed based on the large language model:

[0238] For example, here is an example of a large language model input template refreshed by a policy instruction:

[0239] The task is to generate better policy instructions based on the following information.

[0240] Current time:

[0241] $curr_time

[0242] User issue:

[0243] $query

[0244] Previous strategy instructions:

[0245] $strategy_instruction

[0246] Previous reply:

[0247] $response

[0248] Analysis of errors in previous replies:

[0249] $analysis

[0250] In addition to the considerations included in the previous strategy instructions, and combining the problems with previous responses described in the error analysis, we can analyze what unique challenges and difficulties exist in answering this question, and explore response strategies to improve the usability, readability, and security of responses.

[0251] Based on error analysis, the previous policy instructions were modified.

[0252] The final generated format is: Customized response strategy: ...

[0253] After S305, return to S301 and repeat S301, S302, S304 and S305 until a Y that passes the check is generated.

[0254] Let the original input be X, the strategy instruction be Z, the response be Y, the error analysis report be E, and let n represent the execution round, where n is an integer greater than or equal to 1.

[0255] The policy instruction refresh process of one embodiment of this application can be represented as: X, Z_n, Y, E->Z_(n+1).

[0256] Specifically, in one embodiment, the process of S301 can be represented as: X, Z_n -> Y_n.

[0257] The process of S304 can be represented as: Y_n->E_n.

[0258] The process of S305 can be represented as: X, E_n->Z_(n+1).

[0259] When a response passes the check, it is considered valid data.

[0260] Therefore, in one embodiment, if the check result in S302 is Y, the model is subjected to SFT based on the response that the check passed, and the original input and policy instruction corresponding to the response.

[0261] Specifically, in one embodiment, the response that passes the check, as well as the original input and policy instructions that generate the response, are directly used as training data.

[0262] That is, if the check result in S302 is Y and the check passes, then S303 is executed.

[0263] S303, the training module 204 uses the response to the check and the corresponding original input and policy instructions as fine-tuning training data, and performs SFT on the model based on the fine-tuning training data.

[0264] According to the model training method of this application, the ability to generate responses based on policy instructions is injected into the model during the model training phase. The model training method of this application can improve the quality of the responses generated by the model and avoid errors that may occur during the response generation process.

[0265] Furthermore, in another implementation, instead of directly using the checked response and generating the corresponding original input and policy instructions to train the model, atomic instructions are generated based on the checked response and the corresponding original input and policy instructions to generate atomic instructions that conform to the training data, and the model is then subjected to SFT based on the atomic instructions that conform to the training data.

[0266] Figure 4 is a schematic diagram of a model training device according to an embodiment of this application.

[0267] As shown in Figure 4, the model training device 400 includes:

[0268] The strategy instruction generation module 401 is used to generate strategy instructions based on the original input. The strategy instruction generation module 401 can refer to the strategy instruction generation module 201.

[0269] Inference module 402 is used to input the raw input and policy instructions into the model and obtain the model's output response conforming to the policy instructions. Inference module 402 can be compared with inference module 202.

[0270] The inspection module 403 is used to inspect the quality of the responses obtained by the reasoning module 402 and generate inspection results, including whether the response passes the inspection or fails the inspection. Furthermore, when a response fails the inspection, the inspection module 403 also generates an error analysis report E. The inspection module 403 can refer to the inspection module 203.

[0271] The policy instruction generation module 401 is also used to regenerate policy instructions based on the error analysis report generated by the inspection module 403 and the original input.

[0272] The disassembly module 405 is used to disassemble the response that passes the check, along with the corresponding original input and policy instructions, into atomic compliance items to form atomic instruction compliance training data when the response passes the check.

[0273] Training module 404 is used to perform SFT on the model according to the training data based on atomic instructions.

[0274] Figure 5 shows a flowchart of a model training method according to an embodiment of this application.

[0275] The model training device 400 shown in Figure 4 executes the following process as shown in Figure 5 to achieve model training.

[0276] S500, the policy instruction generation module 401 generates policy instructions based on the original input. Refer to S300.

[0277] S501, the inference module 402 inputs the original input and policy instructions into the model and obtains the model's output response conforming to the policy instructions. See S301.

[0278] S502, the inspection module 403 checks the quality of the response obtained by the reasoning module 402 and generates an inspection result, wherein the inspection result includes whether the response passes the inspection or fails the inspection. Refer to S302.

[0279] If the check result in S502 is a failure, then S504 is executed.

[0280] S504, the inspection module 403 generates an error analysis report for responses that fail the inspection. See S304.

[0281] S505, the policy instruction generation module 401 generates new policy instructions based on the error analysis report and the original input. See S305.

[0282] After S505, return to S501 and repeat S501, S502, S504 and S505 until a response that passes the check is generated.

[0283] In S502, when the response passes the check, S503 is executed.

[0284] S503, the disassembly module 405 performs corresponding atomic compliance item disassembly on the original input and policy instructions corresponding to the checked response to form atomic instruction compliance training data.

[0285] In one embodiment, S503 includes:

[0286] Combining the user question from the original input with the search information obtained from searching based on the user question, several atomic capability tasks are extracted from the search information obtained from searching based on the user question. Atomic capability tasks are tasks related to the search information; however, they are not identical to the user question.

[0287] After extracting the atomic capability tasks, the corresponding atomic capability task responses are summarized based on the search information.

[0288] The atomic capability tasks and their corresponding atomic capability task responses are used as atomic instruction conformation training data.

[0289] Specifically, in one embodiment, the atomic capability task is input into a large neural network model, and the atomic capability task response output by the large neural network model is obtained.

[0290] With atomic capability mission X R The atomic capability mission responded with Y. R , where i is an integer greater than or equal to 1.

[0291] In S503, the final result is of the form X. R _i->Y R The atomic instructions of _i follow the training data.

[0292] Specifically, in one embodiment, the following requirements are followed during the extraction of atomic capabilities task:

[0293] (1) Relevance requirement: Atomic capability tasks must be relevant to the user problem, but cannot completely overlap with or be similar to the user problem or policy instructions;

[0294] (2) Quality requirements: Refer to the strategy instructions to generate atomic capability tasks of higher quality and simpler.

[0295] (3) Diversity requirement: Encourage the generation of diverse atomic capability tasks and ensure that atomic instructions comply with the diversity of data.

[0296] Furthermore, in another embodiment, in S503, after extracting the atomic capability task, the atomic policy instruction corresponding to the atomic capability task is also obtained. Based on the search information, the atomic capability task response corresponding to the atomic capability task is summarized according to the atomic capability task and the atomic policy instruction corresponding to the atomic capability task.

[0297] The atomic capability tasks, the corresponding atomic policy instructions for the atomic capability tasks, and the corresponding atomic capability task responses are used as atomic instruction compliance training data.

[0298] With atomic capability mission X R With atomic strategy instructions as Z R The atomic capability mission responded with Y. R , where i is an integer greater than or equal to 1.

[0299] In S503, the final result is of the form X. R _i, Z R _i->Y R The atomic instructions of _i follow the training data.

[0300] Specifically, in one embodiment, in S503, referring to the implementation of S500, atomic policy instructions are generated based on the atomic capability task.

[0301] Specifically, in another embodiment, in S503, the policy instructions corresponding to the original input are simplified based on the atomic capability task, and the atomic policy instructions corresponding to the atomic capability task are obtained.

[0302] S506 is executed after S503.

[0303] S506, Training module 404 performs SFT on the model according to the atomic instructions and data.

[0304] According to the model training method of this application embodiment, atomic instruction compliance training data is formed based on the original input and policy instructions corresponding to the checked response. By training the model with atomic instruction compliance training data, the model's compliance with policy instructions can be improved, the quality of the responses generated by the model can be enhanced, and errors that may occur during the response generation process can be avoided more effectively.

[0305] Furthermore, in one embodiment, fine-tuning training data and atomic instruction compliance data are obtained, and the model is subjected to SFT based on the fine-tuning training data and atomic instruction compliance data.

[0306] That is, in S502, when the response passes the check, the training module 404 will also use the checked response, along with the corresponding original input and policy instructions, as fine-tuning training data. In S506, the training module 404 performs SFT on the model based on the fine-tuning training data and the atomic instruction compliance data.

[0307] Figure 6 shows a schematic diagram of model training data input according to an embodiment of this application.

[0308] As shown in Figure 6, a policy instruction (X->Z) is generated based on the original input. A response (X, Z->Y) is generated based on the original input and the policy instruction. Finally, atomic capability tasks, atomic policy instructions, and atomic capability task responses (X, Z->X) are obtained based on the original input and the policy instruction. R _i, Z R _i->Y R _i).

[0309] Finally, based on (X, Z->Y) and (X) R _i, Z R _i->Y R _i) Train the model.

[0310] The following is an example illustrating the generation of fine-tuned training data and atomic instruction compliance data based on the original input.

[0311] In one embodiment, the original input includes:

[0312] User issue:

[0313] How many performances were held in City A in the first half of 2024?

[0314] And, the search information obtained based on the user's question:

[0315] [1] Webpage Title: The scene was explosive!! After 10 years, City A was ignited again, XXX revisited the place......||| Webpage Time: 2024-02-24 14:40:00||| Webpage Category: None||| Webpage Content: On February 24th, XXX held a tour concert in City A…

[0316] [2] Webpage Title: "Voice of the Soul" - A City Symphony Concert on March 14, 2024 - Diary ||| Webpage Time: 2024-03-14 23:17:56 ||| Webpage Category: Social ||| Webpage Content: On the evening of March 14, the XXXX Symphony Orchestra held the "Voice of the Soul" symphony concert at the A City National Theater. I was fortunate enough to…

[0317] [3] Webpage Title: XX's New Song Release_Canghai Zixiao_May 12, 2024_Headline|||Webpage Time: 2024-05-12 09:11:00|||Webpage Category: News|||Webpage Content: At 20:30 on May 12, 2024, XX will hold a new song release concert at the People's Gymnasium in City A. …

[0318] [4] Webpage Title: Statistics on Performance Ticket Consumption in City A in the First Half of 2024 Released; Is the Performing Arts and Entertainment Industry Rebounding Again? ||| Webpage Time: 2024-07-15 11:20:00 ||| Webpage Category: None ||| Webpage Content: City A, as a well-known coastal city in China, has a profound cultural heritage, attracting many artists to hold music performances here. ... ||| Keywords: Consumption. The City A Consumption Statistics Bureau today…

[0319] [5]...

[0320] Based on the original input, the policy instructions can be obtained:

[0321] When replying, please note that the search results may only show some of the performances held in City A in the first half of 2024. When replying to users, you can describe the performance in detail, but you need to indicate that the information is not complete.

[0322] Based on the policy instructions and the original input, the response is obtained:

[0323] According to online records, at least three performances were held in City A during the first half of 2024, occurring on February 24, 2024, March 14, 2024, and May 12, 2024. It should be noted that online records may be incomplete, and other performances may have also been held in City A during the first half of 2024…

[0324] Furthermore, the task of extracting atomic capabilities from the original input:

[0325] Regarding the user questions and search results mentioned above, please analyze whether the web pages [1][2][3] in the search results describe the same type of performance.

[0326] Based on the original input, the atomic capability task response _i for atomic capability task R_i is summarized as follows:

[0327] Analysis of the performance times mentioned in search results [1][2][3] reveals that [1][3] are both pop music concerts, belonging to the same type of performance, while [2] is a symphony concert…

[0328] An embodiment of this application also provides a reasoning method and a reasoning apparatus for implementing the reasoning method.

[0329] Figure 7 is a schematic diagram of the structure of a reasoning device according to an embodiment of this application.

[0330] As shown in Figure 7, the inference device 600 includes:

[0331] The policy instruction generation module 601 is used to generate policy instructions based on the original input. The policy instruction generation module 601 can refer to the policy instruction generation module 201.

[0332] Inference module 602 is used to input the raw input and policy instructions into the model and obtain the model's output response conforming to the policy instructions. Inference module 602 can be compared with inference module 202.

[0333] Figure 8 shows a flowchart of a reasoning method according to an embodiment of this application.

[0334] The inference device 600 shown in Figure 7 executes the following process as shown in Figure 8 to achieve model inference.

[0335] S700, the strategy instruction generation module 401 generates strategy instructions based on the original input.

[0336] Specifically, in one embodiment, the strategy instruction generation module 601 calls the first model, inputs the original input into the first model, and obtains the strategy instructions output by the first model.

[0337] That is, the first model is used to implement (X->Z).

[0338] S701, the inference module 602 inputs the original input and policy instructions into the model and obtains the model's output response that conforms to the policy instructions.

[0339] According to the model reasoning method of this application embodiment, introducing strategy instructions during the reasoning process to guide the generation of responses can improve the quality of the responses generated by the model and avoid errors that may occur during the response generation process.

[0340] Specifically, in one embodiment, the inference module 602 calls the second model, inputs the original input and policy instructions into the second model, and obtains the response of the second model that conforms to the policy instructions.

[0341] That is, the second model is used to implement (X, Z->Y).

[0342] Specifically, in one implementation, the first model and the second model are the same model. That is, (X->Z) and (X, Z->Y) share the same parameter model, and the policy instruction serves as the chain of thought (cot) before the response is generated.

[0343] In another implementation, the first model is a model independent of the second model; that is, (X->Z) relies on an external model and is independent of the main model used to generate the response.

[0344] Furthermore, in another embodiment, policy instructions are not generated based on the original input during inference. Instead, generic policy instructions are pre-generated before inference. During inference, the pre-generated generic policy instructions are invoked, and the original input and the invoked policy instructions are input into the model to obtain the model's output response conforming to the policy instructions.

[0345] Alternatively, multiple (or sets of) policy instructions can be pre-generated before inference. During inference, policy instructions (or sets of policy instructions) that match the original input are called from the pre-generated policy instructions. The original input and the called policy instructions are then input into the model, and the model outputs a response that conforms to the policy instructions.

[0346] The model training and inference methods implemented in this application can significantly improve the quality dimensions of the model's inference response, such as information content, security, and logicality, by training the model on only 1 / 10 of the corpus with minor adjustments based on ordinary instructions.

[0347] Figure 9 shows a comparison of the model inference response quality according to an embodiment of this application.

[0348] No policy instruction (X->Y) is introduced during the SFT process, and 60K+ training samples are used to generate model A for SFT.

[0349] In the SFT process, policy instructions (X, Z->Y) are introduced according to the model training method provided in the embodiments of this application, and 3K+ training samples are used to generate model B through SFT.

[0350] Let YA represent the response generated by model A in the model usage scenario. Let YA represent the response generated by model B in the model usage scenario.

[0351] For both Model A and Model B, the security capabilities and value levels of the responses were examined, and the pass rate, failure rate, and critical failure rate were obtained. The final examination results are shown in Figure 9.

[0352] As shown in Figure 9, when the training sample size is reduced to only 5% of that without introducing policy instructions (X->Y), model B, which is fine-tuned based on policy instructions (X, Z->Y), reduces the severe failure rate from 9.73% to 0.99%.

[0353] The method provided in this application is applied to a computing device. This application does not limit the type of computing device that can implement the method proposed in this application. Those skilled in the art can select the appropriate computing device to implement the method provided in this application according to the application scenario requirements.

[0354] For example, in one embodiment, the device can be implemented using a terminal device and / or a server device. The terminal device can be any terminal device capable of interacting with the user, such as a mobile phone, tablet computer, laptop computer, desktop computer, augmented reality (AR) device, VR device, etc.

[0355] In the description of the embodiments of this application, for the sake of convenience, the device is described by dividing it into various modules according to its functions. The division of each module is only a logical functional division. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0356] Specifically, the apparatus proposed in this application can be fully or partially integrated onto a single physical entity (e.g., a GPU or other type of processor), or it can be physically separated. These modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the detection module can be a separate processing element or integrated into a chip in a computing device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together or implemented independently. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0357] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). Alternatively, these modules can be integrated together as a system-on-a-chip (SOC).

[0358] In one embodiment, the model training device and / or model inference device provided in this application can be implemented on a computing device, which includes all the modules of the model training device and / or model inference device provided in this application.

[0359] An embodiment of this application also proposes a computing device. This computing device is used to execute the method flow or part of the method flow described in the embodiments of this application.

[0360] Figure 10 is a schematic diagram of a computing device structure according to an embodiment of this application.

[0361] As shown in Figure 10, the computing device 2400 includes a bus 2404, a processor 2401, a memory 2402, and a communication interface 2403. The processor 2401, the memory 2402, and the communication interface 2403 communicate with each other via the bus 2404.

[0362] The computing device 2400 may be a server or a terminal device. It should be understood that this application does not limit the number of processors or memories in the computing device 2400.

[0363] It is understood that the structural description of the computing device 2400 in the embodiments of this application does not constitute a specific limitation on the computing device 2400. In other embodiments of this application, the computing device 2400 may include other components besides the processor 2401 and the memory 2402.

[0364] Bus 2404 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 10, but this does not imply that there is only one bus or one type of bus. Bus 2404 can include pathways for transmitting information between various components of computing device 2400 (e.g., memory 2402, processor 2401, communication interface 2403).

[0365] Processor 2401 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0366] The processor 2401 may be an on-chip device (SOC) that may include a central processing unit (CPU) and may further include other types of processors.

[0367] The processor 2401 may include, for example, a CPU, DSP, microcontroller, or digital signal processor, and may also include a GPU, embedded neural network processing units (NPUs), and image signal processors (ISPs). The processor may also include necessary hardware accelerators or logic processing hardware circuitry, such as an ASIC, or one or more integrated circuits for controlling the execution of the program in this application. Furthermore, the processor may have the function of operating one or more software programs, which may be stored in a storage medium.

[0368] Processor 2401 may include one or more processing units. For example, a processor may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors. In some embodiments, computing device 2400 may also include one or more processors 2401. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0369] In some embodiments, the processor 2401 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. The USB interface is a USB standard-compliant interface, specifically a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc. The USB interface can be used to connect a charger to charge the computing device, and can also be used for data transfer between the computing device and peripheral devices.

[0370] The memory 2402 may include volatile memory, such as random access memory (RAM). The processor 2401 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0371] The memory 2402 stores executable program code, and the processor 2401 executes the executable program code to implement the aforementioned functions, thereby implementing the method proposed in the embodiments of this application. That is, the memory 2402 stores instructions for executing the method proposed in the embodiments of this application.

[0372] The memory 2402 may include a code storage area and a data storage area. The code storage area may store the operating system. The data storage area may store data created during the use of the computing device 2400. Furthermore, the memory 2402 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage components, flash memory components, universal flash storage (UFS), etc.

[0373] The memory 2402 may be a read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), or other types of dynamic storage devices capable of storing information and instructions. It may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. Alternatively, it may be any computer-readable medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer.

[0374] Processor 2401 and memory 2402 can be combined into a single processing device, but more commonly they are separate components.

[0375] The communication interface 2403 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 2400 and other devices or communication networks.

[0376] The computing device 2400 may also include an external memory interface for connecting an external memory card, such as a Micro SD card, to expand the storage capacity of the computing device. The external memory card communicates with the processor 2401 through the external memory interface to perform data storage functions.

[0377] In another embodiment, the apparatus provided in this application can be implemented on multiple computing devices. For example, the apparatus provided in this application can be implemented through a terminal device and a cloud server device connected to the terminal device.

[0378] An embodiment of this application also proposes a computing device cluster. The computing device cluster includes at least one computing device (which may be a terminal device or a server device), each computing device including a memory and a processor; the processor of the at least one computing device in the computing device cluster is used to execute instructions stored in the memory of the at least one computing device in the computing device cluster, so that the computing device cluster performs the method described in the embodiment of this application.

[0379] Figure 11 is a schematic diagram of a computing device cluster according to an embodiment of this application.

[0380] As shown in Figure 11, the computing device cluster includes at least one computing device 2500 (the structure of computing device 2500 can be referenced to computing device 2400). Computing device 2500 can be a server or a terminal device. Each computing device 2500 includes: a bus 2504 (refer to bus 2404), a processor 2501 (refer to processor 2401), a memory 2502 (refer to memory 2402), and a communication interface 2503 (refer to communication interface 2403).

[0381] The memory 2501 of one or more computing devices in the computing device cluster may contain the same instructions for performing the methods proposed in the embodiments of this application.

[0382] In some possible implementations, the memory 2501 of one or more computing devices 2500 in the computing device cluster may also store partial instructions for executing the glasses try-on method proposed in the embodiments of this application. In other words, a combination of one or more computing devices 2500 can jointly execute instructions for executing the method proposed in the embodiments of this application.

[0383] It should be noted that the memory 2501 in different computing devices 2500 in the computing device cluster can store different instructions, which are used to execute some functions of the model training device proposed in the embodiments of this application.

[0384] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0385] Figure 12 is a schematic diagram of a computing device cluster network connection according to an embodiment of this application.

[0386] As shown in Figure 12, the two computing devices 2600A and 2600B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. The structures of computing devices 2600A and 2600B can be referenced from computing device 2400.

[0387] The computing device 2600A includes: a bus 2604A (refer to bus 2404), a processor 2601A (refer to processor 2401), a memory 2602A (refer to memory 2402), and a communication interface 2603A (refer to communication interface 2403).

[0388] The computing device 2600B includes: a bus 2604B (refer to bus 2404), a processor 2601B (refer to processor 2401), a memory 2602B (refer to memory 2402), and a communication interface 2603B (refer to communication interface 2403).

[0389] In one embodiment, computing device 2600A is a terminal device, and computing device 2600B is a server.

[0390] It should be understood that the functions of computing device 2600A shown in Figure 12 can also be performed by multiple computing devices. Similarly, the functions of computing device 2500B can also be performed by multiple computing devices.

[0391] Figure 13 is a schematic diagram of a system architecture according to an embodiment of this application.

[0392] As shown in Figure 13, the system consists of data collection devices (e.g., data collection devices 1310 and 1311), server devices (e.g., server device 1300), and client devices (e.g., client devices 1320 and 1321). The data collection devices, server devices, and client devices are connected via a communication network.

[0393] The data collection device obtains the data (corpus, etc.) required in the method flow described in the embodiments of this application through various means (manual input, network search, etc.), and transmits it to the server device. The results obtained by the server device after executing the method flow described in the embodiments of this application can be output to the client device for the next natural language processing application.

[0394] In this context, the tasks of data collection devices and client devices can also be directly completed by server devices.

[0395] An embodiment of this application also provides an electronic chip. This electronic chip is used to execute the method flow or part of the method flow described in the embodiments of this application.

[0396] Specifically, the electronic chip includes a processor for executing program instructions. When the computer program instructions are executed by the processor, the electronic chip is triggered to perform the steps described in the embodiments of this application. The processor of the electronic chip may refer to the processor of the computing device described above.

[0397] The devices, apparatuses, and modules described in the embodiments of this application can be implemented by computer chips or physical entities, or by products with certain functions.

[0398] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0399] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0400] Specifically, one embodiment of this application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to execute the method provided in the embodiment of this application.

[0401] An embodiment of this application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform the method provided in the embodiment of this application.

[0402] The embodiments described in this application are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0403] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0404] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0405] It should also be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0406] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0407] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0408] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0409] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments of this application can be implemented using electronic hardware, computer software, or a combination of electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0410] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0411] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A model training method, characterized in that, The method includes: generating a first policy instruction based on an original input, wherein the original input includes a user question, the first policy instruction is related to the original input, the first policy instruction is used to guide the generation of a first response, the first response being a response to the original input; inputting the original input and the first policy instruction into a model, obtaining the first response output by the model, wherein the first response conforms to the first policy instruction; checking the quality of the first response, generating a first check result, wherein the first check result includes whether the first response passes the check or fails the check; if the first response passes the check, performing supervised fine-tuning of the model based on the original input, the first policy instruction, and the first response.

2. The method according to claim 1, characterized in that, The method further includes: when the first response fails the check, generating an error analysis report for the first response; based on the error analysis report, generating a second strategy instruction according to the original input, wherein the second strategy instruction is related to the original input and is used to guide the generation of a second response, the second response being a response to the original input; inputting the original input and the second strategy instruction into the model to obtain the second response output by the model, wherein the second response conforms to the second strategy instruction; checking the quality of the second response and generating a second check result, wherein the second check result includes whether the second response passes the check or fails the check; when the second response passes the check, performing supervised fine-tuning based on the original input, the second strategy instruction, and the second response.

3. The method according to claim 1, characterized in that, The step of generating the first policy instruction based on the original input includes: constructing the first policy instruction for the original input based on the basic principles that the model should follow when performing the task.

4. The method according to claim 3, characterized in that, The basic principles to be followed when performing tasks based on the model, and the construction of the first policy instruction for the original input, include: the basic principles to be followed when performing a first type of task based on the model, and the construction of a policy instruction for the original input, wherein the task type of the first type of task is the task type corresponding to the original input.

5. The method according to claim 1, characterized in that, The first strategy instruction is a natural language instruction used to improve the quality of the first response and avoid errors that may occur during the generation of the first response.

6. The method according to any one of claims 1-5, characterized in that, The supervised fine-tuning based on the original input, the first policy instruction, and the first response includes: using the original input, the first policy instruction, and the first response as fine-tuning training data; and performing supervised fine-tuning on the model based on the fine-tuning training data.

7. The method according to any one of claims 1-5, characterized in that, The original input also includes search information, which is information obtained by searching based on the user's question.

8. The method according to claim 7, characterized in that, The supervised fine-tuning based on the original input, the first policy instruction, and the first response includes: combining the user question in the original input with search information to extract atomic capability tasks from the search information; based on the search information, summarizing atomic capability task responses corresponding to the atomic capability tasks; and performing supervised fine-tuning on the model based on the atomic capability tasks and the atomic capability task responses.

9. The method according to claim 7, characterized in that, The supervised fine-tuning based on the original input, the first policy instruction, and the first response includes: combining the user question in the original input and search information to extract atomic capability tasks from the search information; obtaining atomic policy instructions corresponding to the atomic capability tasks; obtaining atomic capability task responses corresponding to the atomic capability tasks based on the search information, the atomic capability tasks, and the atomic policy instructions; and performing supervised fine-tuning on the model based on the atomic capability tasks, the atomic policy instructions, and the atomic capability task responses.

10. The method according to claim 9, characterized in that, The step of obtaining the atomic policy instruction corresponding to the atomic capability task includes: simplifying the first policy instruction based on the atomic capability task to obtain the atomic policy instruction.

11. A model reasoning method, characterized in that, The method includes: generating a first policy instruction based on an original input; inputting the original input and the first policy instruction into a first model, and obtaining a first response output by the first model, wherein the first response conforms to the first policy instruction.

12. The method according to claim 11, characterized in that, The step of generating the first strategy instruction based on the original input includes: inputting the original input into the second model and obtaining the first strategy instruction output by the second model.

13. The method according to claim 11, characterized in that, The step of generating the first policy instruction based on the original input includes: inputting the original input into the first model and obtaining the first policy instruction output by the first model.

14. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, and each computing device includes a memory and a processor; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as claimed in any one of claims 1-10 or any one of claims 11-13.

15. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computer system, perform the method as claimed in any one of claims 1-10 or any one of claims 11-13.

Images