Model training method and device and computing device cluster

Abstract

The model training method comprises the steps that first training data from a client side are received, and the first training data comprise a first question and a first answer of the first question; reasoning the first problem by adopting a plurality of reasoning schemes through the first model to obtain a plurality of reasoning results; based on the first answer and the multiple reasoning results, at least one preferred reasoning scheme and at least one preferred reasoning result are obtained through screening, and one preferred reasoning scheme is at least associated with one preferred reasoning result; and training the first model based on the first problem, the optimal reasoning scheme and the optimal reasoning result. According to the method, the manpower and material resource cost for marking the training data can be saved. Meanwhile, the model reasoning capability and the average reasoning efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a model training method, apparatus, and computing device cluster. Background Technology

[0002] Large language models perform well in solving most reasoning tasks, such as mathematical reasoning and symbolic reasoning. When using large language models, guiding the model to solve the problem step-by-step, rather than directly providing the answer, usually yields better results. Chain-of-thoughts (CoT) reasoning is an effective approach, guiding the model to demonstrate the reasoning process step-by-step before providing the final answer. In addition, several other approaches can guide the model to output similar reasoning processes, such as the least-to-most approach and the program-of-thoughts (PoT) approach. The least-to-most approach breaks down the problem into several sub-problems and solves them one by one; the program-of-thoughts approach solves the logical part of the problem by writing code and then executing the code to solve the computational part.

[0003] In related technologies, to enable large language models to adopt the aforementioned inference scheme, the inference process for each inference task in the training data is typically labeled during the training phase using manual annotation or strong model annotation. Then, the model is trained using the labeled training data. Finally, the trained model is used to perform inference on new tasks. While this training method effectively improves the model's inference ability and enables it to infer according to a given scheme, it also has limitations: once the model is trained, it can only use a fixed inference scheme. If a different inference scheme is required, the training data needs to be re-labeled and the model retrained, which increases cost and workload. Summary of the Invention

[0004] This application provides a model training method, apparatus, computing device cluster, computer storage medium, and computer product, which can reduce the cost and workload of model training.

[0005] In a first aspect, this application provides a model training method, comprising: receiving first training data from a client, the first training data including: a first question and a first answer to the first question; reasoning about the first question using a first model and employing multiple reasoning schemes to obtain multiple reasoning results; selecting at least one preferred reasoning scheme and at least one preferred reasoning result based on the first answer and the multiple reasoning results, wherein a preferred reasoning scheme is associated with at least one preferred reasoning result; and training the first model based on the first question, the preferred reasoning scheme, and the preferred reasoning result.

[0006] In this way, during model training, the model's own output is used to select the optimal reasoning solutions and results relevant to the problem. These optimal reasoning solutions and results are then used as training data, eliminating the need to label the steps of the reasoning solutions during model training, thus saving the manpower and material costs of labeling training data. Furthermore, different problems within the selected data can have different optimal reasoning solutions and results, enabling the trained model to select appropriate reasoning solutions based on its own capabilities, thereby improving reasoning ability and average reasoning efficiency.

[0007] In one possible implementation, based on a first answer and multiple reasoning results, at least one preferred reasoning scheme and at least one preferred reasoning result are selected. This includes: scoring the reasoning results among the multiple reasoning results based on the first answer to obtain a score for the reasoning result; if the score of the first reasoning result meets the requirements, the reasoning scheme corresponding to the first reasoning result is selected as the preferred reasoning scheme, and the first reasoning result is also selected as the preferred reasoning result, where the first reasoning result is any one of the multiple reasoning results; if the score of each of the multiple reasoning results does not meet the requirements, the reasoning scheme that refuses to answer is selected as the preferred reasoning scheme, and the second reasoning result is selected as the preferred reasoning result, where the second reasoning result is used to represent that the answer cannot be obtained. In this way, preferred reasoning schemes and preferred reasoning results can be selected through scoring. Furthermore, since preferred reasoning schemes can be schemes that refuse to answer, and preferred reasoning results can be responses that refuse to answer, the trained model can directly refuse to answer questions that are beyond its capabilities, saving this part of the reasoning cost and improving the average reasoning efficiency. This also improves the honesty of the model to a certain extent and enhances the user experience.

[0008] In one possible implementation, the number of optimal inference results associated with the optimal inference scheme is below a preset threshold. This prevents a particular problem from appearing too frequently in subsequent training data, reducing data duplication, thereby increasing data diversity and improving model training performance.

[0009] In one possible implementation, a first model is used to reason about the first problem using multiple inference schemes to obtain multiple inference results. This includes: using the first model and the first inference scheme, performing multiple inferences on the first problem to obtain multiple different inference results related to the first inference scheme, where the first inference scheme is any one of the multiple inference schemes. In this way, multiple inference results can be obtained through each inference scheme, increasing the amount of training data.

[0010] In one possible implementation, the first model is specified by the user. This allows users to choose the model they need, improving the user experience.

[0011] In one possible implementation, after training the first model, the process further includes: receiving a second question from the client; using the trained first model to reason about the second question; and returning the reasoning result to the client. This allows the trained model to be used for reasoning on new tasks.

[0012] Secondly, this application provides a model training apparatus, comprising: a communication module and a processing module. The communication module is configured to receive first training data from a client, the first training data including: a first question and a first answer to the first question. The processing module is configured to use a first model and multiple inference schemes to reason about the first question, obtaining multiple inference results. The processing module is further configured to, based on the first answer and the multiple inference results, select at least one preferred inference scheme and at least one preferred inference result, wherein a preferred inference scheme is associated with at least one preferred inference result. The processing module is further configured to train the first model based on the first question, the preferred inference scheme, and the preferred inference result.

[0013] In one possible implementation, when the processing module obtains at least one preferred reasoning scheme and at least one preferred reasoning result based on the first answer and multiple reasoning results, it specifically performs the following: based on the first answer, it scores the reasoning results included in the multiple reasoning results to obtain a score for the reasoning result; if the score of the first reasoning result meets the requirements, it takes the reasoning scheme corresponding to the first reasoning result as the preferred reasoning scheme and the first reasoning result as the preferred reasoning result, wherein the first reasoning result is any one of the multiple reasoning results; if the score of each reasoning result in the multiple reasoning results does not meet the requirements, it takes the reasoning scheme that refuses to answer as the preferred reasoning scheme and the second reasoning result as the preferred reasoning result, wherein the second reasoning result is used to represent that the answer cannot be obtained.

[0014] In one possible implementation, the number of preferred inference results associated with the preferred inference scheme is less than a preset threshold.

[0015] In one possible implementation, when the processing module performs reasoning on the first problem using the first model and employing multiple reasoning schemes to obtain multiple reasoning results, it is specifically used to: perform multiple reasoning operations on the first problem using the first model and employing the first reasoning scheme to obtain multiple different reasoning results related to the first reasoning scheme, wherein the first reasoning scheme is any one of the multiple reasoning schemes.

[0016] In one possible implementation, the first model is specified by the user.

[0017] In one possible implementation, the communication module is also used to receive a second question from the client after the processing module has completed training the first model;

[0018] The processing module is also used to infer the second problem using the trained first model and return the inference result to the client.

[0019] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0020] Fourthly, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device, perform the method described in the first aspect or any possible implementation thereof; or, when executed by a cluster of computing devices, the cluster of computing devices performs the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0021] Fifthly, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the method described in the first aspect or any possible implementation thereof; or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, a cluster of computing devices may include one or more computing devices.

[0022] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0023] Figure 1This is a schematic diagram of the architecture of a model training system provided in an embodiment of this application;

[0024] Figure 2 This is a schematic flowchart of a model training method provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of a model training process provided in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of a model training system that selects a specific model and training data, provided in an embodiment of this application.

[0027] Figure 5 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0031] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0032] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0033] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0034] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0035] Generally, to reduce training costs and improve model inference capabilities, in-context learning (ICL) can be used for inference. In ICL, examples can be provided to guide the model in learning how to solve problems. These examples, as part of the context, help the model understand the structure of the problem and the reasoning process. Specifically, relevant examples can be designed as prompts for the new task; then, the prompts and the new task are input into a large language model for inference. Although this approach allows for flexible changes to the inference scheme and requires only a small number of manually designed examples, it also has limitations: the model's inference performance is weaker than supervised training methods, and the cost of each inference is higher. These limitations arise because this approach does not require training the model but relies on the model's own context learning ability and the quality of the example design. When the model's context learning ability is weak or the designed examples have low similarity to the target task, the inference performance is weak. Furthermore, this approach requires inputting the prompts (examples) and the task into the model for each inference, significantly increasing the model's input length and inference cost.

[0036] In view of this, embodiments of this application provide a model training system. Using this system, during model training, the system can filter out the inference schemes required for the problems in the training dataset by utilizing the model's own output, and use the filtered inference schemes as training data. This eliminates the need to label the steps of the inference schemes during model training, thereby saving the manpower and material costs of labeling training data. Furthermore, different problems in the filtered training data can have different inference schemes, which allows the trained model to select appropriate inference schemes based on its own capabilities, improving inference ability and average inference efficiency.

[0037] For example, Figure 1 A schematic diagram of the architecture of a model training system provided in an embodiment of this application is shown. Figure 1 As shown, the model training system includes a client 100 and a server 200. The client 100 and the server 200 can establish a communication connection via, but not limited to, a wired or wireless network.

[0038] Client 100 can be a desktop application, mobile application, web application, or web-based application, and can be deployed on electronic devices such as mobile phones and computers. Client 100 is configured with a remote training module 110 and an online inference module 120. The remote training module 110 primarily provides an entry point for uploading training data to prepare the data used for model training. Training data can be supervised data with a complete inference process, or semi-supervised data containing only questions and answers but lacking the inference process; there is no limitation here. Supervised data with a complete inference process corresponds to the full inference process in the remote training module 110; semi-supervised data lacking the inference process corresponds to the self-optimization of the inference process in the remote training module 110. Furthermore, the remote training module 110 can also provide an entry point for model selection, allowing users to choose the model they wish to train. The remote training module 110 can also provide an entry point for selecting or inputting inference instances, allowing users to choose inference instances themselves. The online inference module 120 primarily provides an entry point for uploading inference data to prepare the data used for model inference, thereby applying the trained model for inference. For example, client 100 may provide a user interface (UI) so that users can interact with client 100, such as uploading training data, inference data, selecting a model, and selecting / inputting inference instances. The user interface may be a graphical user interface (GUI), a command-line interface (CLI), or other forms of interactive interface; no limitation is made here.

[0039] The server 200 can be, but is not limited to, a server or cloud server, or any device or platform that can provide at least computing resources. The server 200 is configured with an application programming interface (API) 210, a fine-tuning training module 220, an inference module 230, a neural network model 240, a data management module 250, a model self-optimization module 260, and a data optimization module 270. The API 210 is primarily responsible for data transmission, such as receiving data from the client 100 and returning inference results to the client 100. The fine-tuning training module 220 is mainly used to provide online training services, such as fine-tuning the neural network model 240. The inference module 230 is mainly used to provide online inference services, such as performing inference using the trained neural network model 240. The neural network model 240 can be a pre-trained, general-purpose model that can be fine-tuned on various tasks to adapt to specific application scenarios. For example, the neural network model 240 can be understood as, but is not limited to, a foundation model. At least one neural network model 240 is configured in the server 200. When multiple neural network models 240 are configured, users can select the desired model on the client 100 according to their needs. The data management module 250 is mainly used for the orderly management of data used for training and inference.

[0040] The model self-optimization module 260 is mainly used to optimize the model based on the generated results, thereby achieving self-improvement. The model self-optimization module 260 may include: an inference scheme selection module 261, an example design module 262, an inference interface 263, a data optimization interface 264, and a fine-tuning training interface 265. The inference scheme selection module 261 is mainly used to select several suitable inference schemes from a plurality of pre-configured or user-selected inference schemes for each inference task. These inference schemes often require different inference steps and resource consumption. The example design module 262 is mainly used to design an appropriate number of examples for each inference scheme, utilizing the context learning capability of the neural network model to enable the model to output according to the inference scheme of the examples. The inference interface 263 is mainly used to change the generated results of the neural network model for the same problem by sampling decoding and adjusting the difficulty (or order) of examples, thereby enriching data diversity. It can read the problem in the inference task and the examples designed by the example design module 262, generate a prompt, and send the prompt to the neural network model 240. The data optimization interface 264 is mainly used to filter data generated by the model using different inference schemes as training data. The fine-tuning training interface 265 is mainly used to fine-tune the neural network model 240 using the training data filtered by the data optimization interface 264.

[0041] The data selection module 270 primarily assists the data selection interface 264 in the model self-optimization module 230 in filtering data to select training data for model optimization. The data selection module 270 may include a result evaluation module 271 and a data construction module 272. The result evaluation module 271 primarily evaluates the data generated by the neural network model 240 using different inference schemes (i.e., output results (such as inference scheme + inference result)) based on the standard answer (the answer in the user-uploaded data), assigning a reasonable score to each output result. The data construction module 272 primarily determines, based on the score of each data point, whether to package (e.g., add prompts, require the use of a specific inference scheme, or automatically select an inference scheme), select, discard, or modify (e.g., change the answer to a question to "reject answer") before using it as training data.

[0042] In this embodiment, the logical (working / connection) relationship between client 100 and server 200 is as follows: Client 100 sends user-uploaded training data to server 200 via API 210. The fine-tuning training module 220 on server 200 uses the training data to fine-tune the system's default model or the model selected by the user. During the fine-tuning training process, the model self-optimization module 260 and data optimization module 270 on server 200 can filter out the training data required for fine-tuning. After training, client 100 can send user-uploaded inference data to server 200 via API 210. After receiving the inference data from client 100, the inference module 230 on server 200 can execute the model inference process using the trained neural network model. After inference is completed, server 200 can return the inference result to client 100 via API 210.

[0043] The above is an introduction to the model training system provided in the embodiments of this application. The following describes a model training method based on this model training system.

[0044] For example, Figure 2 This diagram illustrates a flowchart of a model training method provided in an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, the method can be executed by a model training device, which can be implemented in software and / or hardware, and can be configured in, but is not limited to, the aforementioned server 200; typically, it can be configured in a cloud computing platform. For ease of description, the server 200 will be used as the execution entity in the following description. Figure 2 As shown, the model training method may include the following steps:

[0045] S201, Server 200 receives first training data from Client 100, wherein the first training data includes a first question and a first answer to the first question.

[0046] In this embodiment, when model training is required, the user can upload their prepared first training data to the client through the data upload portal provided by the client 100. The first training data may include a first question and a first answer to the first question. The client 100 can then transmit the first training data to the server 200. For example, the question may be a problem requiring an answer or explanation.

[0047] S202 and server 200 use the target model and M (M≥2) reasoning schemes to reason about the first problem, and use each reasoning scheme to reason about the first problem K times, so as to obtain MxK reasoning results.

[0048] In this embodiment, after receiving the first training data from the client 100, the server 200 can perform inference on the first problem using the target model. During the inference process, the server 200 can use the target model and employ M inference schemes to infer the first problem, and use each inference scheme to infer the problem K times, obtaining M x K inference results. One inference scheme corresponds to K inference results. The hyperparameters in the target model can be adjusted so that the target model can use the same inference scheme multiple times to obtain different inference results, thereby improving data diversity. It should be understood that the M inference schemes can be preset in the server 200 or uploaded by the user through the client 100; the specific choice depends on the actual situation and is not limited here. In this embodiment, one inference scheme can be associated with one inference example, and a certain inference example can be used to guide the target model to use the inference scheme associated with that inference example. After selecting a certain inference scheme, the inference example corresponding to that inference scheme can be used to guide the model to perform inference. Additionally, the target model can be a model with context learning capabilities, allowing it to learn knowledge from the reasoning scheme. The target model can be a default model on server 200 or a model selected by the user from server 200 via client 100. For example, the reasoning scheme can include two or more of the following: thought chain, simple-to-complex reasoning, procedural thought chain, tree-of-thoughts (ToT), graph-of-thoughts (GoT), and skeleton-of-thoughts (SoT). The reasoning scheme can refer to a series of strategies and methods used to solve problems or make decisions. A reasoning example is a specific instance or scenario demonstrating how logical reasoning, analytical methods, or thought processes can be applied to solve problems or draw conclusions.

[0049] S203, Server 200, based on the answer to the first question and the MxK reasoning results of the first question, selects at least one preferred reasoning scheme and at least one preferred reasoning result, wherein a preferred reasoning scheme is associated with at least one preferred reasoning result.

[0050] In this embodiment, after the server 200 obtains M x K reasoning results for the first question, it can filter the preferred reasoning schemes and preferred reasoning results based on the answer to the question and the M x K reasoning results to obtain at least one preferred reasoning scheme and at least one preferred reasoning result. Specifically, it can score the reasoning results included in the M x K reasoning results based on the first answer to obtain a score for the reasoning result. For example, the reasoning result can be scored based on the similarity between the first answer and the reasoning result. Then, for any reasoning result, when the score of the reasoning result meets the requirements, the reasoning scheme that obtained the reasoning result can be regarded as the preferred reasoning scheme; at the same time, the reasoning result can be regarded as the preferred reasoning result. Since each reasoning scheme has K reasoning results, each preferred reasoning scheme can have at least one preferred reasoning result, that is, a preferred reasoning scheme is associated with at least one preferred reasoning result. For example, if M = 2 and K = 2, reasoning the first question using reasoning scheme 1 can yield reasoning results 1 and 2, and reasoning the first question using reasoning scheme 2 can yield reasoning results 3 and 4. Furthermore, assuming that the similarity between reasoning result 1 and the answer to the first question meets the requirement, the similarity between reasoning result 3 and the answer to the first question also meets the requirement, and the similarity between reasoning result 4 and the answer to the first question also meets the requirement, then reasoning schemes 1 and 2 can be considered as preferred reasoning schemes, and reasoning results 1, 3, and 4 can be considered as preferred reasoning results. It should be understood that when selecting preferred reasoning schemes, for any given reasoning scheme, no more than T (T≤K) of the K reasoning results obtained through that scheme can be selected to control the frequency of the same question appearing in the subsequent total dataset, thus improving data diversity. In other words, the number of preferred reasoning results associated with each preferred reasoning scheme is below a preset threshold, i.e., there is an upper limit. For example, the simpler a question is, the more preferred reasoning results it yields. If the number of selected reasoning results is not controlled, the data repetition for that question will be very high in the subsequent total dataset, which reduces data diversity.

[0051] Furthermore, when the scores of all MxK reasoning results for the first question fail to meet the requirements, it indicates that the question exceeds the capability of the target model. In this case, the reasoning result for the question can be set to "I don't know," indicating that the target model cannot obtain the answer. Simultaneously, the reasoning solution that refuses to answer is designated as the preferred reasoning solution for that question. In this way, when the target model is subsequently trained using this data, it can refuse to answer questions that are beyond its capability. For example, the M reasoning solutions may not include the reasoning solutions that refuse to answer. For example, the preferred reasoning solution for a question can be used to instruct the model to use that preferred reasoning solution to reason about the question.

[0052] S204. Server 200 obtains a target training dataset based on the first problem, the preferred reasoning scheme, and the reasoning result associated with the preferred reasoning scheme. The target training dataset includes at least one second training dataset. A second training dataset includes: the first problem, a preferred reasoning scheme, and a reasoning result associated with the preferred reasoning scheme.

[0053] In this embodiment, after selecting the preferred reasoning scheme and preferred reasoning result for the first problem, the server 200 can obtain a target training dataset based on the first problem, the preferred reasoning scheme, and the reasoning result associated with each preferred reasoning scheme. The target dataset includes at least one second training data set. A second training data set includes the first problem, a preferred reasoning scheme, and a reasoning result associated with that preferred reasoning scheme. The training data can be considered a combination of "problem-preferred reasoning scheme-reasoning result". Specifically, when a preferred reasoning scheme for the first problem is associated with multiple reasoning results, the target dataset contains multiple second training data sets. For example, when the preferred reasoning scheme for problem 1 is preferred reasoning scheme A, and preferred reasoning scheme A is associated with reasoning results B1 and B2, the target dataset may include two training data sets: Problem 1 - Preferred Reasoning Scheme A - Reasoning Result B1, and Problem 1 - Preferred Reasoning Scheme A - Reasoning Result B2.

[0054] S205 and server 200 use the target dataset to train the target model.

[0055] In this embodiment, after obtaining the target training dataset, the server 200 can use the target training dataset to train the target model to obtain the required model. Furthermore, after training the model, the trained model can be used for inference on a new task.

[0056] In this way, during model training, the optimal reasoning solutions and results relevant to the problem are selected by utilizing the target model's own output. These optimal reasoning solutions and results are then used as training data, eliminating the need to label the steps of the reasoning solutions during model training, thus saving the manpower and material costs of labeling training data. Furthermore, different problems in the selected data can have different optimal reasoning solutions and results, allowing the trained model to choose appropriate reasoning solutions based on its own capabilities, improving reasoning ability and average reasoning efficiency. In addition, when answers to questions in the training data are marked as "I don't know," the trained model can directly refuse to answer questions beyond its capabilities, saving the cost of this part of the reasoning and improving average reasoning efficiency. Simultaneously, it enhances the honesty aspect of the model's 3H principles (helpfulness, honesty, and harmlessness) to a certain extent, improving the user experience.

[0057] To understand more vividly Figure 2 The method described below is... Figure 2 To illustrate the model training methods in more vivid terms.

[0058] For example, such as Figure 3As shown, the model training process can include four steps. Step 1: Dataset and Model Selection. This involves determining the target model and the target dataset (i.e., the training dataset). The target model should be a large language model with context learning capabilities, and the target dataset should be a reasoning task with questions and answers, but it doesn't necessarily need to include the reasoning process (although it can). Step 2: Model Inference. The target model generates multiple outputs based on different reasoning examples (few-shot examples). This can be achieved by sampling to obtain multiple sets of outputs for the same input. In related technologies, detailed data training with inference processes is required to improve the model's reasoning ability. However, this approach uses the inference process generated by the model itself as training data, simplifying the data labeling process. Step 3: Training Data Optimization. The results generated by the model in the previous step are scored based on the standard answers in the target dataset, and the results with higher scores are selected as training data. If the target model's answer to a question is incorrect under multiple reasoning schemes and multiple sampling conditions, the answer to that question is marked as "I don't know" (i.e., directly rejecting the answer) and added to the training data. In this way, the model can be trained to select reasoning schemes and refuse to answer questions beyond its capabilities. Step 4: Model Training. The target model is fine-tuned based on the data selected in the previous step to obtain the aligned model. Further, after completing model training, model inference can be performed, i.e., step: model inference, using the trained model to perform inference on a new task.

[0059] To facilitate understanding of the effectiveness of the model trained using the scheme provided in the embodiments of this application, examples are given below.

[0060] For example, such as Figure 4 As shown, Mistral-7B was selected as... Figure 2 The target model in (soon) Figure 1 The neural network model 240 shown is replaced with the language model Mistral-7B, and experiments are conducted using the GSM8K dataset. The training set portion of GSM8K, after removing the inference process, is used as remote training data provided by the user, and the test set portion of GSM8K is used as the target scene. The final comparison results between the scheme provided in this application embodiment and the scheme using context-based learning capabilities are shown in Table 1 below.

[0061]

[0062]

[0063] Table 1

[0064] As can be seen from Table 1 above:

[0065] There are three metrics in total: accuracy, input tokens, and output tokens. Accuracy is the most important metric, measuring the strength of reasoning ability. The number of input and output tokens represents reasoning efficiency; the fewer input and output tokens used, the higher the reasoning efficiency.

[0066] Test results of the untrained model with prompt: The accuracies of the four different inference schemes are 10.31, 36.85, 40.64 and 41.02 respectively, with the best result being 41.02;

[0067] Test results of the model trained using the method provided in the embodiments of this application: If four different inference schemes are specified, the accuracies are 9.93, 43.37, 44.20 and 56.86, respectively. If the model is required to automatically select the appropriate scheme, the accuracy is 48.98.

[0068] As can be seen from Table 1 above, using the method provided in this application embodiment, which allows the model to autonomously select an inference scheme (i.e., Auto in the inference method), compared with the best-performing method in the baseline, the relative accuracy is improved to 19.4%.

[0069] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments or technical features involved in the embodiments described above can be combined according to the actual situation, and the combined solution is still within the protection scope of this application.

[0070] Based on the methods in the above embodiments, this application also provides a model training device.

[0071] For example, Figure 5 A schematic diagram of the structure of a model training device provided in an embodiment of this application is shown. Figure 5As shown, the model training device 500 includes a communication module 501 and a processing module 502. The communication module 501 receives first training data from a client, the first training data including a first question and a first answer to the first question. The processing module 502 uses the first model and multiple inference schemes to perform inference on the first question, obtaining multiple inference results. The processing module 502 is further configured to, based on the first answer and multiple inference results, select at least one preferred inference scheme and at least one preferred inference result, wherein each preferred inference scheme is associated with at least one preferred inference result. The processing module 502 is also used to train the first model based on the first question, the preferred inference scheme, and the preferred inference result.

[0072] In some embodiments, when the processing module 502 obtains at least one preferred reasoning scheme and at least one preferred reasoning result based on the first answer and multiple reasoning results, it is specifically used to: score the reasoning results included in the multiple reasoning results based on the first answer to obtain a score for the reasoning results; if the score of the first reasoning result meets the requirements, take the reasoning scheme corresponding to the first reasoning result as the preferred reasoning scheme and take the first reasoning result as the preferred reasoning result, wherein the first reasoning result is any one of the multiple reasoning results. If the score of each of the multiple reasoning results does not meet the requirements, take the reasoning scheme that refuses to answer as the preferred reasoning scheme and take the second reasoning result as the preferred reasoning result, wherein the second reasoning result is used to characterize that the answer cannot be obtained.

[0073] In some embodiments, the number of preferred inference results associated with a preferred inference scheme is less than a preset threshold.

[0074] In some embodiments, when the processing module 502 performs reasoning on the first problem using a first model and employing multiple reasoning schemes to obtain multiple reasoning results, it is specifically used to: perform multiple reasoning operations on the first problem using a first model and employing a first reasoning scheme to obtain multiple different reasoning results related to the first reasoning scheme, wherein the first reasoning scheme is any one of the multiple reasoning schemes.

[0075] In some embodiments, the first model is specified by the user.

[0076] In some embodiments, the communication module 501 is further configured to receive a second question from the client after the processing module 502 has completed training the first model. The processing module 502 is further configured to use the trained first model to perform reasoning on the second question and return the reasoning result to the client.

[0077] In some embodiments, Figure 5Both the communication module 501 and the processing module 502 shown can be implemented in software or in hardware. For example, the implementation of the communication module 501 will be described below. Similarly, the implementation of the processing module 502 can be referenced from the implementation of the communication module 501.

[0078] As an example of a software functional unit, the communication module 501 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the communication module 501 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0079] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0080] As an example of a hardware functional unit, the communication module 501 may include at least one computing device, such as a server. Alternatively, the communication module 501 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0081] The multiple computing devices included in the communication module 501 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the communication module 501 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the communication module 501 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0082] It should be noted that, in other embodiments, the communication module 501 can be used to execute any step in the model training method described in the above embodiments, and the processing module 502 can be used to execute any step in the model training method described in the above embodiments. Furthermore, the steps implemented by the communication module 501 and the processing module 502 can be specified as needed, and different steps in the model training method described in the above embodiments can be implemented by the communication module 501 and the processing module 502 respectively. Figure 5 The model training device 500 shown has all the functions.

[0083] This application also provides a computing device 600. For example... Figure 6 As shown, the computing device 600 includes a bus 602, a processor 604, a memory 606, and a communication interface 608. The processor 604, the memory 606, and the communication interface 608 communicate with each other via the bus 602. The computing device 600 can be a server or an electronic device. It should be understood that this application does not limit the number of processors and memories in the computing device 600.

[0084] Bus 602 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus 604 is represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 604 may include a path for transmitting information between various components of the computing device 600 (e.g., memory 606, processor 604, communication interface 608).

[0085] Processor 604 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0086] Memory 606 may include volatile memory, such as random access memory (RAM). Processor 604 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0087] The memory 606 stores executable program code, and the processor 604 executes the executable program code to implement the aforementioned functions. Figure 5 The communication module 501 and processing module 502 shown herein perform their functions to implement the model training method described in the above embodiments. That is, the memory 606 stores instructions for executing the model training method described in the above embodiments.

[0088] Alternatively, the memory 606 stores executable code, and the processor 604 executes the executable code to implement the aforementioned functions respectively. Figure 5 The model training apparatus 500 shown in the diagram performs the functions of the model training method described in the above embodiments. That is, the memory 606 stores instructions for executing the model training method described in the above embodiments.

[0089] The communication interface 603 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 600 and other devices or communication networks.

[0090] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be an electronic device such as a desktop computer, a laptop computer, or a smartphone.

[0091] like Figure 7As shown, the computing device cluster includes at least one computing device 600. The memory 606 of one or more computing devices 600 in the computing device cluster may store the same instructions for executing the model training method described in the above embodiments.

[0092] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing the model training method described in the above embodiments. In other words, a combination of one or more computing devices 600 can jointly execute instructions for executing the model training method described in the above embodiments.

[0093] It should be noted that the memory 606 in different computing devices 600 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 5 The model training device 500 shown contains some of the functions. That is, the instructions stored in the memory 606 in different computing devices 600 can implement the functions of one or more modules in the communication module 501 and the processing module 502.

[0094] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 8 One possible implementation is shown. For example... Figure 8 As shown, two computing devices 600A and 600B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 606 in computing device 600A stores instructions for executing the functions of communication module 501. Simultaneously, the memory 606 in computing device 600B stores instructions for executing the functions of processing module 502.

[0095] It should be understood that Figure 8 The functions of computing device 600A shown can also be performed by multiple computing devices 600. Similarly, the functions of computing device 600B can also be performed by multiple computing devices 600.

[0096] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 7 and Figure 8 The connection method of the computing device cluster is different in that the memory 606 of one or more computing devices 600 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0097] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, a combination of one or more computing devices 600 can jointly execute the instructions for executing the aforementioned data processing method.

[0098] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the methods in the above embodiments; or, when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices performs the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0099] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the methods in the above embodiments, or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the methods in the above embodiments.

[0100] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0101] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0102] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0103] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method comprises: receiving first training data from a client, the first training data comprising: a first question and a first answer to the first question; reasoning on the first question by a first model and using a plurality of reasoning schemes to obtain a plurality of reasoning results; based on the first answer and the plurality of reasoning results, screening at least one preferred reasoning scheme and at least one preferred reasoning result, wherein one of the preferred reasoning schemes is associated with at least one of the preferred reasoning results; training the first model based on the first question, the preferred reasoning scheme and the preferred reasoning result.

2. The method of claim 1, wherein, The method comprises: based on the first answer, scoring the reasoning results contained in the plurality of reasoning results to obtain scores of the reasoning results; in the case that the score of a first reasoning result meets the requirement, taking the reasoning scheme corresponding to the first reasoning result as the preferred reasoning scheme, and taking the first reasoning result as the preferred reasoning result, wherein the first reasoning result is any one of the plurality of reasoning results; in the case that the scores of all the reasoning results in the plurality of reasoning results do not meet the requirement, taking a reasoning scheme for refusing to answer as the preferred reasoning scheme, and taking a second reasoning result as the preferred reasoning result, wherein the second reasoning result is used to represent that the answer cannot be obtained.

3. The method according to claim 1 or 2, characterized in that, The number of the preferred reasoning results associated with the preferred reasoning scheme is lower than a preset threshold.

4. The method according to any of claims 1 to 3, characterized in that, The method comprises: by the first model, reasoning on the first question using a first reasoning scheme to obtain a plurality of different reasoning results related to the first reasoning scheme, wherein the first reasoning scheme is any one of the plurality of reasoning schemes.

5. The method according to any of claims 1 to 4, characterized in that, The first model is specified by a user.

6. The method according to any one of claims 1 to 5, characterized in that, After the training of the first model is completed, the method further comprises: receiving a second question from the client; using the trained first model to reason on the second question and returning the reasoning result to the client.

7. A model training apparatus characterized by comprising: The method comprises: a communication module configured to receive first training data from a client, the first training data comprising: a first question and a first answer to the first question; a processing module configured to reason on the first question by a first model and using a plurality of reasoning schemes to obtain a plurality of reasoning results; the processing module is further configured to screen at least one preferred reasoning scheme and at least one preferred reasoning result based on the first answer and the plurality of reasoning results, wherein one of the preferred reasoning schemes is associated with at least one of the preferred reasoning results; the processing module is further configured to train the first model based on the first question, the preferred reasoning scheme and the preferred reasoning result.

8. The apparatus of claim 7, wherein, The processing module is specifically configured to: score the reasoning results included in the plurality of reasoning results based on the first answer to obtain scores of the reasoning results; in a case where the score of the first reasoning result meets a requirement, take a reasoning scheme corresponding to the first reasoning result as the preferred reasoning scheme, and take the first reasoning result as the preferred reasoning result, wherein the first reasoning result is any one of the plurality of reasoning results. in a case where the scores of the reasoning results in the plurality of reasoning results all do not meet the requirement, take a reasoning scheme for refusing to answer as the preferred reasoning scheme, and take a second reasoning result as the preferred reasoning result, wherein the second reasoning result is used to represent that the answer cannot be obtained.

9. The apparatus of claim 7 or 8, wherein, The number of the preferred reasoning results associated with the preferred reasoning scheme is lower than a preset threshold.

10. The apparatus of any of claims 7-9, wherein, The processing module is specifically configured to: use the first model and a plurality of reasoning schemes to perform reasoning on the first question to obtain a plurality of reasoning results.

11. The apparatus of any of claims 7-10, wherein, The first model is specified by a user.

12. The apparatus of any of claims 7-11, wherein, The communication module is further configured to receive a second question from the client after the processing module completes training of the first model. The processing module is further configured to use the trained first model to perform reasoning on the second question and return a reasoning result to the client.

13. A cluster of computing devices, characterized in that, The at least one computing device includes a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method of any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by the computing device cluster, cause the computing device cluster to perform the method of any one of claims 1-6, wherein the computing device cluster includes at least one computing device.

15. A computer program product comprising instructions, characterized in that, The computer program instructions, when executed by the computing device cluster, cause the computing device cluster to perform the method of any one of claims 1-6, wherein the computing device cluster includes at least one computing device.

Images