Model updating method and model updating system

Abstract

The invention discloses a model updating method and a model updating system, and the method comprises the steps: carrying out the reasoning of reasoning data through a draft model, obtaining a plurality of reasoning results, inputting the reasoning data and the plurality of reasoning results into a target model, enabling the target model to carry out the reasoning of the reasoning data, and carrying out the verification of the plurality of reasoning results. Then, according to a verification result of the target model, determining a reasoning result accepted by the target model from the multiple reasoning results, according to the number of the accepted reasoning results, obtaining an acceptance rate of the reasoning data, and when the acceptance rate is smaller than or equal to a threshold value, caching the reasoning data as first training data; and when a condition for updating the draft model is detected, performing model training by using the first training data, and updating the draft model by using a training result. The draft model is updated according to the reasoning data with the poor reasoning performance, so that the accuracy and pertinence of draft model updating are improved, the acceptance rate of the reasoning result output by the updated draft model is increased, and the speculation reasoning performance is improved.

Description

Technical Field

[0001] This application relates to the field of information technology (IT), and more particularly to a model updating method and a model updating system. Background Technology

[0002] Speculative inference is a method to accelerate model inference. Speculative inference uses two model parameters, a draft model and a target model, with significantly different parameters. The draft model is used for inference of simple parts, while the target model is used for inference of complex parts, thereby reducing the number of autoregressions in the target model and accelerating the overall inference performance.

[0003] The performance of speculative reasoning is limited by how closely the draft model approaches the target model. When the draft model is weak, the acceptance rate of the tokens output by the draft model is low, which leads to an increase in the number of autoregressions in the target model and a decrease in overall reasoning performance, thus preventing the advantages of speculative reasoning from being realized.

[0004] Therefore, how to improve the capabilities of draft models remains to be studied. Summary of the Invention

[0005] This application provides a model update method and system to increase the acceptance rate of tokens output by the draft model and improve speculative inference performance.

[0006] Firstly, this application provides a model update method, which can be applied to a device, a module of the device (such as a chip), or a system corresponding to the device. The device can be a terminal device or a network device (such as a server). Based on this, the method includes:

[0007] The draft model is used to infer multiple inference results from the inference data. The inference data and the multiple inference results are then input into the target model. The target model is used to infer from the inference data and verify the multiple inference results. Based on the verification results of the target model, the inference results accepted by the target model are determined from the multiple inference results. The acceptance rate of the inference data is obtained based on the number of accepted inference results. When the acceptance rate is less than or equal to a threshold, the inference data is cached as first training data. Finally, when a condition for updating the draft model is detected, the first training data is used to train the model, and the training results are used to update the draft model.

[0008] In the above scheme, the draft model and the target model are used to perform speculative inference on the inference data. The inference result can be understood as the token output by the draft model. When the acceptance rate of the inference data is less than or equal to a threshold, it indicates that the inference performance of the draft model for that inference data is poor. That is, the token (i.e., the inference result) output by the draft model for that inference data has a low probability of being accepted by the target model. Therefore, this inference data is used as the data (i.e., the first training data) when updating the draft model to improve the accuracy and relevance of the draft model update, thereby increasing the acceptance rate of the token output by the updated draft model and improving the speculative inference performance.

[0009] One possible implementation is that the condition for updating the draft model is that the amount of the first training data reaches a first threshold.

[0010] One possible implementation of the method further includes: acquiring the acceptance rates of multiple inference data; determining a system acceptance rate based on the acceptance rates of the multiple inference data; and updating the draft model when the system acceptance rate is less than a second threshold.

[0011] In the above implementation, the first threshold and the second threshold can be understood as performance detection conditions for the draft model. When the conditions for updating the draft model are met, it indicates that the performance of the draft model is poor. This allows the draft model to be updated when its performance is determined to be poor, thereby optimizing speculative inference and eliminating the need for periodic updates to the draft model. This avoids frequent and ineffective updates and reduces the overhead of computer resources.

[0012] One possible implementation, wherein the step of training the model using the first training data and updating the draft model using the training results, includes: simultaneously inputting the first training data and offline second training data into a first training model and a second training model to obtain a first result output by the first training model and a second result output by the second training model; wherein the first training model corresponds to the draft model, the second training model corresponds to the target model, and both the first training model and the second training model are used for cyclic autoregression until the results of output inference data are output; using the first result as a negative sample and the second result as a positive sample, training the first training model to obtain the model parameters of the first training model; and updating the draft model using the model parameters of the first training model.

[0013] In the above implementation, the first training model can be generated directly using the model parameters of the draft model, and the second training model can be generated directly using the model parameters of the target model. That is, the model parameters of the first training model are the same as those of the draft model, and the model parameters of the second training model are the same as those of the target model. It can be understood that the draft model is configured to perform K autoregressions per inference iteration, and the target model is configured to perform 1 autoregression per inference iteration, where K is an integer greater than 1. In other words, the draft model and the target model will only infer a portion of the inference data each time, including multiple inference results (e.g., the draft model infers K tokens each time). Only after multiple inference iterations of the inference data can the complete result of the inference data be obtained. The first and second training models are both used for iterative autoregression until the result of the inference data is output (or it can be understood that neither the first nor the second training model is configured with an autoregression iteration count; after the inference data is input into the first or second training model, the first or second training model iteratively performs autoregression until the complete result of the inference data is output). That is, the first and second training models directly output the complete result of the inference data in a single inference iteration.

[0014] Therefore, the first result is equivalent to all the true results output by the draft model for the inference data, and the second result is equivalent to all the true results output by the target model for the same inference data, thus ensuring the authenticity of negative and positive samples. Training the first training model with these true training samples ensures its accuracy, which in turn ensures the accuracy of updating the draft model, effectively improving its capabilities. This can increase the acceptance rate of tokens output by the draft model, reduce the number of autoregressions in the target model, decrease the inference latency of speculative inference, and achieve overall performance optimization for speculative inference.

[0015] One possible implementation is that the ratio of the first training data to the second training data is a preset value.

[0016] In the above implementation, by setting the ratio of offline inference data to speculative inference data, the types of inference data used when updating the draft model are balanced, avoiding the use of the same type of inference data, preventing the updated draft model from gradually biasing towards inferring one or a few types of inference data, and improving the generalization and accuracy of the draft model update.

[0017] Secondly, this application provides a model update system, which includes a computing device and a training device. The computing device is used to infer multiple inference results from inference data using a draft model, input the inference data and the multiple inference results into a target model, and the target model is used to infer the inference data and verify the multiple inference results. Based on the verification results, the inference results accepted by the target model are determined from the multiple inference results, and the acceptance rate of the inference data is obtained based on the number of accepted inference results. When the acceptance rate is less than or equal to a threshold, the inference data is cached as first training data.

[0018] The training device is configured to train the model using the first training data and update the draft model using the training results when a condition for updating the draft model is detected.

[0019] One possible implementation is that the condition for updating the draft model is that the amount of the first training data reaches a first threshold.

[0020] In one possible implementation, the computing device is further configured to acquire the acceptance rates of multiple inference data; the processing module is configured to determine the system acceptance rate based on the acceptance rates of the multiple inference data; and the condition for updating the draft model is that the system acceptance rate is less than a second threshold.

[0021] One possible implementation is that the training device is specifically used to: simultaneously input the first training data and offline second training data into a first training model and a second training model to obtain a first result output by the first training model and a second result output by the second training model; wherein, the first training model corresponds to the draft model, the second training model corresponds to the target model, and both the first training model and the second training model are used for cyclic autoregression until the results of the inference data are output; using the first result as a negative sample and the second result as a positive sample, the first training model is trained to obtain the model parameters of the first training model; and the draft model is updated using the model parameters of the first training model.

[0022] One possible implementation is that the ratio of the first training data to the second training data is a preset value.

[0023] Thirdly, this application provides a computer device, including a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface are connected via the bus and communicate with each other. The memory stores computer execution instructions. When the computer device is running, the processor executes the computer execution instructions in the memory to perform the operation steps of the method described in the first aspect or any possible implementation of the first aspect using the hardware resources in the computer device.

[0024] Fourthly, a computer device cluster is provided, comprising a plurality of computer devices as described in the third aspect, each computer device being used independently or jointly to perform the operational steps of the method described in the first aspect or any possible implementation thereof.

[0025] Fifthly, this application provides an apparatus comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program and execute the method provided in the first aspect above.

[0026] In a sixth aspect, this application provides a non-volatile computer-readable storage medium comprising a program that, when executed on a device, causes the device to perform the operational steps of the method described in the first aspect or any possible implementation thereof.

[0027] In a seventh aspect, this application provides a computer program product that, when run on a device, causes the device to perform the operational steps of the method described in the first aspect or any possible implementation thereof.

[0028] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0029] Figure 1 This is a schematic diagram illustrating an application scenario of speculative reasoning.

[0030] Figure 2 A flowchart illustrating a model update method provided in this application;

[0031] Figure 3 A flowchart illustrating a speculative reasoning process provided in this application;

[0032] Figure 4 A flowchart illustrating a model update process provided in this application;

[0033] Figure 5 A schematic diagram of the structure of a device provided in this application;

[0034] Figure 6 This is a schematic diagram of the structure of a device provided in this application. Detailed Implementation

[0035] With the rapid development of Large Language Model (LLM) technology, the scale of model parameters (or simply parameter count) is also increasing. For autoregressive inference LLMs, such as Transformer architecture models, it is believed that models with a large number of parameters perform better than models with a small number of parameters in solving complex problems and handling complex scenarios. However, Transformer architecture models are usually limited by memory bandwidth during the sampling process. For a given hardware, the time required to generate a single token is approximately proportional to the size of the model parameters and the amount of memory.

[0036] It is known that to ensure the quality of model inference, the model needs to output tokens in the order of iteration. That is, for any given inference data, the model infers one token through autoregression, and then, based on that token, performs autoregression again to infer the next token, until all results for that inference data are obtained. Since it's impossible to reduce model inference latency by outputting multiple tokens in parallel, it's understandable that the larger the model's parameter count, the longer it takes to infer a single token, i.e., the longer the latency of one autoregression cycle. Therefore, inferring multiple tokens will result in significant latency consumption for models with a large number of parameters. Here, a token refers to the smallest unit of meaning that the model can understand and generate; it is the basic unit of the model. A token can represent a word, a part of a word, or a character. Tokens are assigned numerical values ​​or identifiers, arranged in sequences or vectors, and input or output from the model; they are the model's linguistic building blocks.

[0037] Therefore, speculative inference techniques are used in related technologies to reduce inference latency for models with large parameter sets. Specifically, speculative inference involves using two models with significantly different parameter sets: the model with fewer parameters (referred to as the draft model or small model) is used for inferring the simpler parts, while the model with more parameters (referred to as the target model or large model) is used for inferring the more complex parts. The draft model is configured to perform K autoregressions, and the target model is configured to perform one autoregression, where K is an integer greater than 1. The speculative inference process is described below.

[0038] First, a draft model performs K autoregressions based on a specified prefix, generating K tokens (i.e., multiple inference results are obtained from a single inference, and each inference result can be understood as a partial result of the inference data), and recording the distribution of the corresponding K tokens. The target model concatenates the K tokens generated by the draft model with the prefix as input, performs one autoregression, and outputs a new token and the distribution of the aforementioned K tokens. This process can be called the target model's validation process. For example, if m tokens are accepted by the target model and Km tokens are rejected, then in this process: the draft model performs K autoregressions, the target model performs 1 autoregression, and a total of m+1 tokens are output. Assuming m is an integer greater than or equal to 1, it can be seen that the target model only needs to perform 1 autoregression to obtain at least 2 tokens. Because the latency of the draft model's autoregression is much smaller than that of the target model's autoregression, the inference of the draft model reduces the number of autoregressions in the target model, thereby achieving an overall acceleration of inference performance.

[0039] Ideally, the target model accepts all K tokens output by the draft model each time. This greatly reduces the number of autoregressions performed by the target model, thus accelerating overall inference performance. For example, the inference data is the sequential output of the 26 English letters. The draft model infers K tokens each time, where K=5. During the first inference, the 5 tokens inferred by the draft model (the initial inference result of the draft model is "a", "b", "c", "d", "e") are all accepted by the target model, which infers one token based on these 5 tokens (the initial inference result of the target model is "abcdef"). It can be seen that the target model can obtain 6 letters in one inference (or in other words, the target model can infer 6 letters in one autoregression). Therefore, the target model only needs to perform 5 autoregressions to obtain the sequentially ordered 26 English letters, eliminating the need for 26 autoregressions.

[0040] However, in practice, the performance of speculative inference is limited by how closely the draft model approximates the target model. When the draft model is weak, the acceptance rate of the tokens it outputs is low, leading to an increase in the number of autoregressions in the target model and a decrease in overall inference performance, thus preventing speculative inference from demonstrating its advantages. Therefore, it is necessary to improve the capabilities of the draft model, that is, to improve the acceptance rate of the tokens it outputs.

[0041] One approach proposes identifying training samples from the inference results of each inference iteration of the draft model, and then updating the draft model based on these results. The inference results include sequence information (e.g., K tokens) and distribution information (which can be understood as the probability distribution of each of the K tokens). For example, inference results where a token is rejected by the target model, inference results with a rejection rate greater than a threshold, or inference results where the number of tokens rejected by the target model exceeds a threshold, are used as training samples.

[0042] However, in the above scheme, for a given inference data, the draft model typically needs to output multiple inference results. The later output inference result is determined based on the previous inference result after validation in the target model. For example, the inference data is: outputting the 26 English letters in sequence. The draft model outputs K tokens each time, where K=5, meaning each inference result includes 5 tokens. Suppose the draft model's first inference result includes the 5 tokens "a", "b", "c", "d", and "q". These 5 tokens, along with the inference data, are then input into the target model for validation. The target model will reject the token "q" and output the token "abcde". Based on this, the draft model's second inference will be based on "e" instead of "q". In other words, because of the target model's validation, the draft model's inference result for the inference data is altered (for ease of description, we can call it a non-true inference result), making it impossible to obtain the draft model's true inference result for the inference data. Therefore, it can be seen that non-realistic reasoning results have low reliability and authenticity as training samples. Using these non-realistic reasoning results as training samples will affect the accuracy of updating the draft model, thus making it difficult to effectively improve the draft model's capabilities. In addition, this approach generally updates the draft model periodically, or when the amount of training sample data reaches a threshold. This may lead to frequent updates, affecting the accuracy of the draft model and wasting computer resources.

[0043] To address this, this application proposes a model update method that updates the draft model based on the actual results of the draft model for inference data, ensuring the accuracy of the draft model update, thereby improving the draft model's capabilities (i.e., increasing the acceptance rate of the tokens output by the draft model), reducing the latency of speculative inference, and improving the performance of speculative inference.

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operating methods in the method embodiments can also be applied to the device embodiments or system embodiments.

[0045] First, the application scenarios to which this application applies will be introduced. (See reference...) Figure 1 As shown, users can send inference data (such as access requests, query requests, translation requests, etc.) to server 120 through a client running on terminal device 110. Server 120 stores a first training model and a second training model. The first training model is generated based on the model parameters of the draft model, and the second training model is generated based on the model parameters of the target model. Both the first and second training models are used for cyclic autoregression until the results of the inference data are output. The draft model and the target model are used for speculative inference on the inference data. Optionally, the draft model and the target model can be deployed on server 120, or they can be deployed in a speculative inference system.

[0046] Server 120 inputs the inference data from the training set into the first training model and the second training model, thereby obtaining the first result corresponding to the inference data output by the first training model and the second result corresponding to the inference data output by the second training model. Then, the first result is used as a negative sample and the second result is used as a positive sample to train the first training model, thereby obtaining the model parameters of the first training model; finally, the draft model is updated based on the model parameters of the first training model.

[0047] The following explanations and descriptions of some technologies and terms involved in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0048] (1) Acceptance rate

[0049] Acceptance rate can be understood as the probability that the inference result output by the draft model based on the inference data is accepted by the target model. In speculative inference, the draft model is used to perform multiple inferences on the inference data, yielding multiple inference results each time. After each inference, the multiple inference results and the inference data are input into the target model so that the target model can perform inferences on the inference data and validate these inference results, thus obtaining the validation result output by the target model. Based on this, the inference result accepted by the target model can be determined from the multiple inference results according to the validation result, and the acceptance rate of the inference data can be calculated based on the number of accepted inference results.

[0050] It is understandable that the draft model performs multiple inferences on the inference data. Similarly, the target model also performs multiple inferences on the multiple inference results obtained by the draft model each time, as well as on the inference data. Therefore, the acceptance rate of the inference data is the average probability that the multiple inference results obtained by the draft model from multiple inferences is accepted by the target model. For example, based on the multiple inference results obtained by the draft model from the i-th inference on the inference data, and the verification results of the multiple inference results obtained by the target model from the i-th inference, the probability that the multiple inference results obtained from the i-th inference is accepted by the target model is calculated, thus obtaining the acceptance rate of the inference data in the i-th inference; then, the average of the n acceptance rates of the inference data is calculated to obtain the acceptance rate of the inference data, where i is an integer greater than or equal to 1, and i is less than or equal to n, and n is the number of times the draft model performs inferences on the inference data. An example is described below.

[0051] For example, the inference data is: outputting the 26 letters of the English alphabet in sequence. The draft model outputs K tokens each time, where K=5. Based on this, suppose the first 5 tokens output by the draft model are "a", "b", "c", "d", and "q". These 5 tokens, along with the inference data, are input into the target model for validation. It can be seen that the target model will reject the token "q" and accept the four tokens "a", "b", "c", and "d". That is, the probability that the first K tokens output by the draft model are accepted by the target model is 80%. Similarly, suppose the draft model performs 7 inferences, and the probabilities that the K tokens output in these 7 inferences are accepted by the target model are 80%, 80%, 60%, 60%, 20%, 40%, and 100%, respectively. Based on this, the acceptance rate for this inference data is approximately 63%.

[0052] One possible implementation is to cache the inference data as first training data when the acceptance rate of the inference data is less than or equal to a threshold; wherein, the threshold can be a value preset based on experience, such as 65%, and this application does not limit the threshold. It can be understood that each piece of inference data less than or equal to the threshold is used as the first training data, that is, the first training data is the inference data obtained through speculative inference by the draft model and the target model.

[0053] Optionally, the first training data is cached in the training set. Therefore, it can be understood that the training set includes at least one inference data, which is the data used to update the draft model. Optionally, all the inference data in the training set is the first training data; optional, a portion of the inference data in the training set is the first training data, and the remaining portion is offline second training data; optional, all the inference data in the training set is offline second training data. For ease of description, the first training data in the training set is referred to as hard example data (or difficult data), and the second training data in the training set is referred to as offline data. The offline data includes inference data for various preset business types; this application does not limit the business types of the offline data.

[0054] One possible implementation involves acquiring a pre-defined set of offline data from an offline dataset based on the number of hard examples, before meeting the conditions for updating the draft model, and then storing this offline data in the training set. For example, when identifying an inference data point with an acceptance rate below a threshold, one offline data point is acquired from the offline dataset, and both data are then stored in the training set.

[0055] One possible implementation involves acquiring offline data from the offline dataset based on the number of hard examples when the conditions for updating the draft model are met, and storing this offline data in the training set. This application does not impose a limitation on the timing of adding offline data to the training set.

[0056] Optionally, the ratio of difficult examples to offline data in the training set is a preset value, that is, the number of difficult examples and the number of offline data meet a preset ratio. This preset value can be a value preset based on experience, or it can be set according to factors such as business type, preset number of offline data, etc., for example, the preset value is 1 (or the preset ratio is 1:1). This application does not limit the preset value here.

[0057] One possible implementation is to reuse the offline data when the condition for updating the draft model is met. If the number of hard examples is greater than the number of offline data in the offline dataset, the offline data can be reused to achieve a 1:1 ratio between hard examples and offline data in the training set. Similarly, if the number of hard examples is less than the number of offline data in the offline dataset, the hard examples can be reused to achieve a 1:1 ratio between hard examples and offline data in the training set. Alternatively, a portion of the offline data can be randomly selected from the offline dataset to achieve a 1:1 ratio between hard examples and offline data in the training set.

[0058] This application balances the business types of the inference data used when updating the draft model by setting the ratio of the first training data to the second training data, avoiding the use of inference data of the same business type, and preventing the updated draft model from gradually becoming biased towards inference data of one or a few business types, thereby improving the generalization and accuracy of the draft model update.

[0059] (2) Conditions for updating the draft model

[0060] The conditions for updating the draft model refer to the conditions (or timing) under which the draft model is updated. For ease of description, it will be referred to as the update conditions below. When the update conditions are met, the draft model will be updated.

[0061] One possible implementation is that the update conditions include quantity conditions and system acceptance rate conditions. When the quantity conditions and / or system acceptance rate conditions are met, the update conditions are determined to be met.

[0062] The quantity condition refers to the condition being met when the number of difficult examples reaches a first threshold, or in other words, when the number of inference data with an acceptance rate less than or equal to the threshold reaches the first threshold. The first threshold can be a value preset based on experience, such as 500; this application does not limit the first threshold.

[0063] The system acceptance rate condition refers to the system acceptance rate being less than or equal to a second threshold. That is, the system acceptance rate condition is met when the system acceptance rate is less than or equal to the second threshold. Here, the system acceptance rate is the average probability of the acceptance rates corresponding to the inference data obtained through speculative inference using the draft model and the target model. The second threshold can be a value preset based on experience, such as 60%. This application does not limit the second threshold here. In other words, the system acceptance rate is determined based on the acceptance rates of all inference data obtained through speculative inference. It can be understood that the system acceptance rate is updated each time the acceptance rate of an inference data point is determined. For example, if the acceptance rate of the first inference data point is 80%, the system acceptance rate is 80%; if the acceptance rate of the second inference data point is 60%, the system acceptance rate is 70%, and so on. This application will not elaborate further on this.

[0064] In summary, the update conditions can be understood as performance testing conditions for the draft model. When the update conditions are met, it indicates that the draft model's performance is poor, or in other words, its speculative inference performance is poor. That is, the update conditions better reflect the performance of the current speculative inference technique, allowing the draft model to be updated when its performance is determined to be poor (e.g., the system acceptance rate is less than the second threshold). This optimizes the speculative inference technique, avoids periodic updates to the draft model, prevents frequent and ineffective updates, and reduces the overhead of computer resources.

[0065] Based on the above information, Figure 2 This application provides a schematic diagram of a model update process. This process is applied to a device, a module of the device (such as a chip), or a system corresponding to the device. The device can be a terminal device or a network device (such as a server). The device stores a first training model and a second training model (or the device can use both). The first training model is generated based on the model parameters of a draft model, and the second training model is generated based on the model parameters of a target model. Both the first and second training models are used for iterative autoregression until the result of the inference data is output. That is, neither the first nor the second training model is configured with an autoregression count. During the iterative autoregression process, the first or second training model outputs the result of the inference data until a termination marker (such as a preset token) is reached. The draft model and the target model are used for speculative inference on the inference data. The speculative inference process includes, for example, the following:

[0066] The process involves acquiring inference data and inputting it into a draft model to obtain the first inference result. Then, the inference data and the first inference result from the draft model are input into the target model to obtain the first inference result. Next, the first inference result from the target model and the inference data are used as input to the draft model to obtain the second inference result. This process is repeated until all results corresponding to the inference data are output. Similarly, after acquiring the next set of inference data, the above process is repeated to perform speculative inference on each set of inference data.

[0067] like Figure 2 As shown, the method may include the following steps:

[0068] Step 201: Use the draft model to infer the inference data to obtain multiple inference results. Input the inference data and multiple inference results into the target model. The target model is used to infer the inference data and verify the multiple inference results. Based on the verification results, determine the inference results accepted by the target model from the multiple inference results, and obtain the acceptance rate of the inference data based on the number of accepted inference results.

[0069] The acceptance rate in this process is described in the above-mentioned content and will not be repeated here.

[0070] Step 202: When the acceptance rate of the inference data is less than or equal to the threshold, cache the inference data as the first training data.

[0071] In this process, the description of the first training data is as described above, and will not be repeated here.

[0072] Step 203: When the condition for updating the draft model is detected, train the model using the first training data and update the draft model using the training results.

[0073] The conditions for updating the draft model in this process are described above and will not be repeated here.

[0074] One possible implementation is to train the model using only the first training data, or only the second training data, or both the first and second training data.

[0075] One possible implementation involves training a first training model, where the training result refers to the model parameters obtained after training (i.e., training convergence). For example, inference data from the training set is input into both the first and second training models to obtain a first result corresponding to the inference data output by the first training model and a second result corresponding to the inference data output by the second training model. Then, the first result is used as a negative sample and the second result as a positive sample to train the first training model, obtaining its model parameters. Finally, the draft model is updated using the model parameters of the first training model. Optionally, the training set includes both first and second training data; for details, refer to the above description, which will not be repeated here.

[0076] The first training model can be generated directly using the model parameters of the draft model, and the second training model can be generated directly using the model parameters of the target model. That is, the model parameters of the first training model are the same as those of the draft model, and the model parameters of the second training model are the same as those of the target model. For example, the model parameters of the draft model can be copied, and the first training model can be generated based on these parameters; or, in other words, the draft model can be copied to obtain the first training model. Training the first training model is equivalent to training the draft model, thus ensuring the accuracy of updates to the draft model.

[0077] In this process, for any inference data in the training set, the inference data is input into the first training model and the second training model respectively, thereby obtaining the first result output by the first training model and the second result output by the second training model. Assuming that the training set includes 100 inference data, then 100 first results output by the first training model and 100 second results output by the second training model will be obtained.

[0078] It's understandable that neither the first nor the second training model is configured with an autoregression count. Therefore, after inputting the inference data into either model, it iterates through the autoregression process until a final marker is reached, at which point it outputs all the results of the inference data. In other words, all the results of the inference data are obtained directly through a single inference iteration. The first result output by the first training model is equivalent to all the inference results output by the draft model for that inference data; these results are not validated by the target model. In other words, the first result can be understood as all the true inference results output by the draft model for that inference data. Similarly, the second result can be understood as all the true inference results output by the target model for that inference data. For example, if the inference data is the sequential output of the 26 English letters, the first result includes the token sequence output by the first training model (equivalent to the draft model) (e.g., the 26-letter sequence "abcdqmno…xyz"), and the second result includes the token sequence output by the second training model (equivalent to the target model) (e.g., the 26-letter sequence "abcdefghijk…xyz"). Therefore, negative samples include the first results corresponding to all inference data in the training set, and positive samples include the second results corresponding to all inference data in the training set.

[0079] Optionally, the training and convergence of the first training model can be achieved by calculating the loss value based on gradient descent, stochastic gradient descent, dynamic learning rate adjustment, or distance calculation function of knowledge distillation. This application does not limit the training process of the first training model.

[0080] In this process, the model parameters of the first training model refer to the model parameters after training. These model parameters include, but are not limited to, weights, slope, bias, and hyperparameters.

[0081] One possible implementation is to directly use the model parameters of the first trained model as the model parameters of the draft model to update the draft model.

[0082] One possible implementation involves updating the draft model based on the model parameters of the first trained model, then resetting the system acceptance rate to conduct the next round of performance testing on the draft model (or speculative inference). The description of the system acceptance rate is as described above and will not be repeated here.

[0083] In summary, in this application, the first result is equivalent to all true inference results of the draft model for the inference data, and the second result is equivalent to all true inference results of the target model for the inference data, thus ensuring the authenticity of negative and positive samples. Training the first training model with these true training samples (including negative and positive samples) ensures the accuracy of the first training model, thereby ensuring the accuracy of updating the draft model and effectively improving the draft model's capabilities. Therefore, it can increase the acceptance rate of the tokens output by the draft model, reduce the number of inferences in the target model, reduce the inference latency of speculative inference techniques, and achieve overall performance optimization of speculative inference techniques.

[0084] To better illustrate the above technical solution, it will be described in two processes: one is the speculative reasoning process, and the other is the model update process.

[0085] Figure 3 A flowchart illustrating a speculative reasoning process provided in this application is shown below. Figure 3 As shown, this speculative reasoning process may include the following steps:

[0086] Step 301: Obtain inference data and perform speculative inference on the inference data based on the target model and the draft model.

[0087] It is understandable that by performing speculative reasoning on the inference data based on the draft model and the target model, multiple inference results of the draft model for the inference data and multiple verification results of the target model for the inference data can be obtained.

[0088] Step 302: Calculate the acceptance rate of the inference data based on the inference results of the draft model and the verification results of the target model.

[0089] In this process, referring to the above introduction on the acceptance rate of inference data, this application will not elaborate on the calculation of the acceptance rate of inference data.

[0090] Step 303: Determine whether the acceptance rate of the inference data is less than or equal to the threshold; if yes, proceed to step 304; otherwise, perform other processing.

[0091] It is understood that step 303 is executed for each piece of inference data used for speculative inference based on the draft model and the target model. Other processing could include ending the process, discarding the inference data, and returning to step 302 to calculate the acceptance rate of the next inference data, etc. This application does not limit the other processing.

[0092] Step 304: Cache the inference data as hard example data in the training set.

[0093] At this point, the inference data is determined to be hard example data, and then the inference data is cached in the training set in the memory (buffer) so that the inference data can be used as the data when updating the draft model.

[0094] Figure 4 A flowchart illustrating a model update process provided in this application is shown below. Figure 4 As shown, this model update process is performed after step 301 above, and may specifically include the following steps:

[0095] Step 401: Calculate the system acceptance rate based on the acceptance rate of the inference data.

[0096] Referring to the above description of the system acceptance rate, this application will not elaborate on the system acceptance rate here.

[0097] Step 402: Determine whether the system acceptance rate is less than the second threshold. If yes, proceed to step 403; otherwise, perform other processing. Other processing may include ending the process and returning to step 401 to update the system acceptance rate based on the acceptance rate of the next inference data. This application does not limit the scope of other processing.

[0098] Step 403: Based on the difficult example data, store the preset offline data into the training set according to the preset ratio.

[0099] When the update condition is met, based on the number of difficult examples in the training set at this time, preset offline data is obtained from the offline dataset according to a preset ratio, and this offline data is stored in the training set, so that the number of difficult examples and offline data in the training set meets the preset ratio. For details regarding the training set, please refer to the above description; this application will not repeat it here.

[0100] Step 404: Input each inference data in the training set into the first training model and the second training model respectively; wherein, the first training model is used to output the corresponding first result for each inference data, and the second training model is used to output the corresponding second result for each inference data.

[0101] Step 405: Train the first training model based on the first result output by the first training model and the second result output by the second training model.

[0102] In this process, the first result corresponding to each inference data output is taken as a negative sample, and the second result corresponding to each inference data output is taken as a positive sample. That is, the first training model is trained based on the negative and positive samples until the first training model converges (e.g., model convergence is achieved based on knowledge distillation). For a detailed description, please refer to step 203 above, which will not be repeated here.

[0103] Step 406: Update the model parameters of the draft model based on the model parameters of the first trained model.

[0104] In this step, the model parameters of the first trained model can be directly used as the model parameters of the draft model. That is, the model parameters of the current draft model are replaced with the model parameters of the first trained model, thus obtaining the updated draft model. It can be understood that the model parameters of the first trained model refer to the model parameters of the first trained model after training in step 405.

[0105] The above text combines Figure 2 , Figure 3 , Figure 4 The method for updating the model according to this application is described in detail below, and will be combined with... Figure 5 , Figure 6 This application describes the apparatus provided for performing the above-described method.

[0106] Figure 5 This is a schematic diagram of a model update device provided in this application. The device 500 can be used to implement the above-described model update method, and therefore can also achieve the beneficial effects of the above-described method embodiments.

[0107] like Figure 5 As shown, the device 500 includes a processing module 510 and a training module 520. The processing module 510 is used to infer multiple inference results from inference data using a draft model, input the inference data and the multiple inference results into a target model, which is used to infer the inference data and verify the multiple inference results. Based on the verification results, it determines the inference results accepted by the target model from the multiple inference results and obtains the acceptance rate of the inference data based on the number of accepted inference results. When the acceptance rate is less than or equal to a threshold, the inference data is cached as first training data. The training module 520 is used to train the model using the first training data when a condition for updating the draft model is detected, and to update the draft model using the training results.

[0108] One possible implementation is that the condition for updating the draft model is that the amount of the first training data reaches a first threshold.

[0109] In one possible implementation, the device 500 further includes an acquisition module 530, which is used to acquire the acceptance rates of multiple inference data; the processing module 510 is used to determine the system acceptance rate based on the acceptance rates of the multiple inference data; and the condition for updating the draft model is that the system acceptance rate is less than a second threshold.

[0110] In one possible implementation, the training module 520 is specifically configured to: simultaneously input the first training data and the offline second training data into the first training model and the second training model to obtain a first result output by the first training model and a second result output by the second training model; wherein, the first training model corresponds to the draft model, the second training model corresponds to the target model, and both the first training model and the second training model are used for cyclic autoregression until the results of the inference data are output; using the first result as a negative sample and the second result as a positive sample, the first training model is trained to obtain the model parameters of the first training model; and the draft model is updated using the model parameters of the first training model.

[0111] One possible implementation is that the ratio of the first training data to the second training data is a preset value.

[0112] The processing module 510, training module 520, and acquisition module 530 can all be implemented in software or in hardware. For example, the implementation of the processing module 510 will be described below. Similarly, the implementation of the training module 520 and acquisition module 530 can refer to the implementation of the processing module 510.

[0113] As an example of a software functional unit, processing module 510 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 510 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0114] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0115] As an example of a hardware functional unit, the processing module 510 may include at least one computing device, such as a server. Alternatively, the processing module 510 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0116] The processing module 510 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the processing module 510 includes multiple computing devices that can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 510 includes multiple computing devices that can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0117] It is understood that in other embodiments, the processing module 510, the training module 520, and the acquisition module 530 can be used for any step in the above model update method.

[0118] It should be noted that the module division in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical entities, or have two or more units integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] Based on the above embodiments, this application also provides an apparatus that can implement the methods in the above embodiments and has the functions of device 500. (See also...) Figure 6 As shown, the device 600 includes a transceiver 601, a processor 602, and a memory 603. The transceiver 601, the processor 602, and the memory 603 are interconnected.

[0121] Optionally, the transceiver 601, the processor 602, and the memory 603 are interconnected via a bus 604. The bus 604 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0122] The transceiver 601 is used to receive and send signals to enable communication with other devices.

[0123] The function of the processor 602 can be referred to the description in the above embodiments, and will not be repeated here.

[0124] The processor 602 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP, etc. The processor 602 may further include hardware chips. These hardware chips can be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. When implementing the above functions, the processor 602 can be implemented in hardware, or it can be implemented by hardware executing corresponding software. The steps of the method disclosed in the above embodiments of this application can be directly reflected as the processor 602 completing the execution, or as the hardware and software modules in the processor 602 combining to complete the execution.

[0125] The memory 603 is used to store program instructions and data. Specifically, the program instructions may include program code, which includes computer operation instructions. The memory 603 may include volatile memory, such as random access memory (RAM); it may also include non-volatile memory, such as at least one disk storage device, hard disk drive (HDD), or solid state drive (SSD). The memory 603 may also be any other medium capable of carrying or storing program code in the form of instructions or data structures that can be accessed by a computer; this application does not limit this. The processor 602 executes the program instructions stored in the memory 603 to implement the above functions, thereby implementing the method provided in the above embodiments.

[0126] Based on the above embodiments, this application also provides a model training system, which includes one or more computing devices and one or more training devices.

[0127] The computing device is used to infer multiple inference results from the inference data using a draft model, input the inference data and multiple inference results into a target model, and the target model is used to infer the inference data and verify the multiple inference results; based on the verification results, the inference results accepted by the target model are determined from the multiple inference results, and the acceptance rate of the inference data is obtained based on the number of accepted inference results; when the acceptance rate is less than or equal to a threshold, the inference data is cached as the first training data.

[0128] The training device is used to train the model using the first training data when a condition for updating the draft model is detected, and to update the draft model using the training results.

[0129] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, causes the computer to perform the methods provided in the above embodiments.

[0130] Optionally, the aforementioned computer may include, but is not limited to, communication devices such as terminal devices and network devices.

[0131] The storage medium can be any available medium that a computer can access. For example, but not limited to, a computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0132] Based on the above embodiments, this application also provides a computer program product, which, when run on a device, causes the device to execute the method provided in the above embodiments.

[0133] Based on the above embodiments, this application also provides a chip for reading a computer program stored in a memory to implement the method provided in the above embodiments. Optionally, the chip may include a processor coupled to the memory for reading the computer program stored in the memory to implement the method provided in the above embodiments. Optionally, the chip may also include components such as a memory, a communication interface, and a power supply module. The memory is used to store the computer program; the communication interface is used to receive and send data; and the power supply module is used to supply power to the processor.

[0134] Based on the above embodiments, this application provides a chip system including a processor for supporting a computer device in implementing the functions involved in the federated learning system in the above embodiments. In one possible design, the chip system further includes a memory for storing necessary programs and data of the computer device. This chip system may be composed of chips or may include chips and other discrete devices.

[0135] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0136] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A model updating method characterized by, The method comprises: inference data using a draft model to obtain a plurality of inference results, inputting the inference data and the plurality of inference results into a target model, the target model being used for inferring the inference data and verifying the plurality of inference results, determining an inference result accepted by the target model from the plurality of inference results according to a verification result, and obtaining an acceptance rate of the inference data according to a number of accepted inference results; when the acceptance rate is less than or equal to a threshold value, caching the inference data as first training data; when a condition for updating the draft model is detected, performing model training using the first training data, and updating the draft model using a training result.

2. The method of claim 1, wherein, The condition for updating the draft model is that the number of the first training data reaches a first threshold value.

3. The method of claim 1, wherein, The method further comprises: obtaining acceptance rates of a plurality of inference data; determining a system acceptance rate according to the acceptance rates of the plurality of inference data; The condition for updating the draft model is that the system acceptance rate is less than a second threshold value.

4. The method of claim 1, wherein, The method further comprises: inputting the first training data and offline second training data into a first training model and a second training model at the same time to obtain a first result output by the first training model and a second result output by the second training model; wherein the first training model corresponds to the draft model, the second training model corresponds to the target model, and the first training model and the second training model are both used for cyclic autoregression until the result of the inference data is output; performing model training on the first training model by taking the first result as a negative sample and the second result as a positive sample to obtain model parameters of the first training model; updating the draft model by using the model parameters of the first training model.

5. The method of claim 4, wherein, The ratio of the first training data to the second training data is a preset value.

6. A model updating system characterized by comprising: The method comprises: a computing device and a training device; The computing device is configured to infer inference data using a draft model to obtain a plurality of inference results, input the inference data and the plurality of inference results into a target model, and infer the inference data and verify the plurality of inference results by using the target model; determine an inference result accepted by the target model from the plurality of inference results according to a verification result, and obtain an acceptance rate of the inference data according to a number of accepted inference results; when the acceptance rate is less than or equal to a threshold value, cache the inference data as first training data; 7. The system of claim 6, wherein, The training device is configured to perform model training using the first training data when a condition for updating the draft model is detected, and update the draft model using a training result.

8. The system of claim 6, wherein, The condition for updating the draft model is that the number of the first training data reaches a first threshold value. The computing device is further configured to: obtain acceptance rates of a plurality of inference data; determine a system acceptance rate according to the acceptance rates of the plurality of inference data; 9. The system of claim 6, wherein, The condition for updating the draft model is that the system acceptance rate is less than a second threshold value. The training device is specifically configured to: inputting the first training data and offline second training data into a first training model and a second training model simultaneously to obtain a first result output by the first training model and a second result output by the second training model; wherein the first training model corresponds to the draft model, the second training model corresponds to the target model, and the first training model and the second training model are both used for cyclic self-recurrence until a result of inference data is output; performing model training on the first training model by taking the first result as a negative sample and the second result as a positive sample to obtain model parameters of the first training model; updating the draft model by using the model parameters of the first training model.

10. The system of claim 9, wherein, The ratio of the first training data to the second training data is a preset value.

11. A non-transitory computer readable storage medium, comprising: The non-volatile computer readable storage medium includes a program which, when running on the device, causes the device to perform the operation steps of any one of the methods in claims 1 to 5.

12. A computer program product, characterised in that, The computer program product, when running on the device, causes the device to perform the operation steps of any one of the methods in claims 1 to 5.

Images