Draft word prediction method and apparatus, device, and storage medium

Abstract

The application relates to the technical field of prediction, and discloses a draft word prediction method, device, equipment and storage medium, the method comprising the following steps: extracting a to-be-predicted input sequence and a low-rank adaptation identifier from a received request, inputting the to-be-predicted input sequence and the low-rank adaptation identifier into a preset draft model for prediction, obtaining a draft word, the preset draft model comprising a gating module and at least two expert modules, the gating module is used for determining a corresponding target expert module in each expert module according to the low-rank adaptation identifier, so that the target expert module predicts the to-be-predicted input sequence to obtain the draft word, and the draft word is input into a target main model for verification. According to the application, the gating module determines the corresponding target expert module in each expert module according to the low-rank adaptation identifier, so that the target expert module predicts the to-be-predicted input sequence to obtain the draft word, and the dynamic probability distribution of the main model after mounting different low-rank adaptation weights can be fitted.

Description

Technical Field

[0001] This application relates to the field of prediction technology, and in particular to a method, apparatus, device and storage medium for predicting draft lexical terms. Background Technology

[0002] Speculative decoding is a technique for accelerating inference in large language models. In the autoregressive generation process of large language models, each generated lexical requires a complete forward propagation, resulting in significant computational overhead and high inference latency. Speculative decoding introduces a draft model with fewer parameters to quickly generate multiple consecutive candidate lexicals. These candidate lexicals are then validated in parallel by the main model, thus merging multiple sequential forward propagations into a single parallel computation, significantly improving inference efficiency.

[0003] Currently, existing speculative decoding schemes have achieved good results in single-master model scenarios. These schemes typically equip a single master model with a corresponding draft model, and train the draft model to fit the output distribution of the master model through distillation. However, in low-rank adaptation (LoRA) inference scenarios, the actual effective weights of the base model change dynamically based on different user requests by loading corresponding low-rank adaptation adapters. This means that even with the same input context, the vocabulary probability distribution of the master model's output will differ significantly after attaching different low-rank adaptation adapters. Existing speculative decoding schemes, equipped only with a single draft model, cannot simultaneously fit the dynamic probability distribution of the master model after attaching different low-rank adaptation weights. Summary of the Invention

[0004] The main purpose of this application is to provide a draft lexical prediction method, apparatus, device and storage medium, which aims to solve the problem that the existing speculative decoding scheme is only equipped with a single draft model, and cannot simultaneously fit the dynamic probability distribution of the main model after attaching different low-rank adaptive weights.

[0005] To achieve the above objectives, this application proposes a draft lexical prediction method, the method comprising: Extract the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request; The input sequence to be predicted and the low-rank adaptation identifier are input into a preset draft model for prediction to obtain draft words. The preset draft model includes a gating module and at least two expert modules. The gating module is used to determine the corresponding target expert module in each of the expert modules according to the low-rank adaptation identifier, so that the target expert module can predict the input sequence to be predicted to obtain the draft words. The draft lexical units are input into the target main model for verification to obtain the text prediction result of the input sequence to be predicted. The target main model is obtained by superimposing the adapter corresponding to the low-rank adaptation identifier.

[0006] In one embodiment, the step of inputting the input sequence to be predicted and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units includes: The corresponding adapter is determined based on the low-rank adaptation identifier, and the adapter is superimposed on the base model to obtain the target main model; The input sequence to be predicted is input into the target main model for forward propagation to generate the hidden layer state information of the main model corresponding to the input sequence to be predicted. The input sequence to be predicted, the hidden layer state information of the main model corresponding to the input sequence to be predicted, and the low-rank adaptation identifier are input into the preset draft model for prediction to obtain draft lexical units.

[0007] In one embodiment, the preset draft model further includes: an embedding module, a projection module, and a decoding sampling module. The step of inputting the input sequence to be predicted, the hidden layer state information of the main model corresponding to the input sequence to be predicted, and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units includes: The embedding module extracts features from the input sequence to be predicted and the hidden layer state information of the main model corresponding to the input sequence to be predicted to obtain fused features. The gating module determines the target expert module corresponding to the low-rank adaptation identifier. The target expert module processes the fused features to obtain the hidden layer state information of the expert module corresponding to the fused features; The projection module maps the hidden layer state information of the expert module corresponding to the fusion feature into a vocabulary probability distribution. The decoding and sampling module obtains draft lexical units by decoding and sampling according to the probability distribution of the vocabulary.

[0008] In one embodiment, prior to the step of extracting the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request, the method includes: Obtain the training corpus sequence and input the training corpus sequence into the target main model for forward propagation to generate the hidden layer state information of the main model and the first probability distribution corresponding to the training corpus sequence; Input the hidden layer state of the main model corresponding to the training corpus sequence into the initial draft model to obtain the second probability distribution; The loss is determined based on the first probability distribution and the second probability distribution, and the initial draft model is updated based on the loss to obtain a preset draft model.

[0009] In one embodiment, the step of inputting the training corpus sequence into the target main model for forward propagation to generate the main model hidden layer state information and a first probability distribution corresponding to the training corpus sequence includes: The training corpus sequence is input into the target main model for forward propagation to generate the hidden layer state information of the main model corresponding to the training corpus sequence and the first probability distribution corresponding to each position to be predicted in the training corpus sequence. The step of inputting the hidden layer state of the main model corresponding to the training corpus sequence into the initial draft model to obtain the second probability distribution includes: The training corpus sequence is gradually shifted to the right to obtain at least two shifted sequences; Based on the hidden layer state of the main model corresponding to the training corpus sequence and the first offset sequence, the second probability distribution of the first position to be predicted is obtained; The second probability distribution of the non-first prediction position is obtained by combining the hidden layer state information of the expert module corresponding to the previous position of the non-first prediction position with the offset sequence corresponding to the non-first prediction position.

[0010] In one embodiment, the step of determining the loss based on the first probability distribution and the second probability distribution, and updating the initial draft model based on the loss to obtain a preset draft model includes: The loss corresponding to each of the predicted locations is determined based on the first probability distribution and the second probability distribution of each of the predicted locations; The total loss is obtained by weighting the losses corresponding to each of the predicted positions according to their respective weights. The weight of the loss corresponding to each predicted position is negatively correlated with the depth of the corresponding predicted position. The initial draft model is updated based on the total loss to obtain a preset draft model.

[0011] In one embodiment, the step of inputting the draft lexical units into the target main model for verification to obtain the text prediction result of the input sequence to be predicted includes: The draft lexical units are grouped in order to obtain several lexical units, and each lexical unit is validated at the same time, with the number of lexical units in each lexical unit increasing sequentially. If there are tuples that fail to be verified in each of the tuples, determine the target tuple that has the smallest number of tuples and fails to be verified, and concatenate the previous tuple of the target tuple with the input sequence to be predicted to obtain a new sequence to be predicted. Return to the step of inputting the input sequence to be predicted and the low-rank adaptation identifier into the preset draft model for prediction to obtain draft words, until there are no failed word groups in each word group, and then concatenate the latest draft word group with the latest input sequence to be predicted as the text prediction result.

[0012] Furthermore, to achieve the above objectives, this application also proposes a draft lexical prediction device, the device comprising: The information extraction module is used to extract the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request. The lexical acquisition module is used to input the input sequence to be predicted and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units. The preset draft model includes a gating module and at least two expert modules. The gating module is used to determine the corresponding target expert module in each of the expert modules according to the low-rank adaptation identifier, so that the target expert module can predict the input sequence to be predicted to obtain the draft lexical units. The lexical verification module is used to input the draft lexical into the target main model for verification, and obtain the text prediction result of the input sequence to be predicted. The target main model is obtained by superimposing the adapter corresponding to the low-rank adaptation identifier.

[0013] In addition, to achieve the above objectives, this application also proposes a draft lexical prediction device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the draft lexical prediction method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the draft lexical prediction method as described above.

[0015] This application provides a draft lexical prediction method, apparatus, device, and storage medium. The method includes: extracting a target input sequence and a low-rank adaptation identifier from a received prediction request; inputting the target input sequence and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units; the preset draft model includes a gating module and at least two expert modules; the gating module is used to determine a corresponding target expert module among the expert modules based on the low-rank adaptation identifier, so that the target expert module predicts the target input sequence to obtain the draft lexical units; inputting the draft lexical units into a target main model for verification to obtain a text prediction result of the target input sequence; the target main model is obtained by superimposing an adapter corresponding to the low-rank adaptation identifier.

[0016] The draft lexical prediction method of this application extracts the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request, and inputs the input sequence to be predicted and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units. The preset draft model includes a gating module and at least two expert modules. The gating module is used to determine the corresponding target expert module among the expert modules according to the low-rank adaptation identifier, so that the target expert module predicts the input sequence to be predicted to obtain the draft lexical units. The draft lexical units are then input into the target main model for verification to obtain the text prediction result of the input sequence to be predicted. Compared with existing speculative decoding schemes that only have a single draft model, which cannot simultaneously fit the dynamic probability distribution of the main model after attaching different low-rank adaptation weights, this application can use the gating module to determine the corresponding target expert module among the expert modules according to the low-rank adaptation identifier, so that the target expert module predicts the input sequence to be predicted to obtain the draft lexical units, thereby fitting the dynamic probability distribution of the main model after attaching different low-rank adaptation weights. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application; Figure 2This is a flowchart illustrating the first embodiment of the word prediction method in the draft of this application; Figure 3 This is a diagram showing the module connection relationships of the preset draft model in the first embodiment of the draft lexical prediction method of this application. Figure 4 This is a flowchart illustrating the overall workflow of the word prediction method in the first embodiment of the draft of this application. Figure 5 This is a flowchart illustrating the second embodiment of the word prediction method in the draft of this application; Figure 6 This is a flowchart illustrating the overall training process in the second embodiment of the word prediction method in the draft of this application. Figure 7 This is a flowchart illustrating the third embodiment of the word prediction method in the draft of this application; Figure 8 This is a structural block diagram of the word prediction device in the draft of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application.

[0023] like Figure 1 As shown, the draft lexical prediction device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may be connected to a display screen; optionally, the user interface 1003 may include a standard wired interface or a wireless interface. In this application, the wired interface of the user interface 1003 may be a USB interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or non-volatile memory (NVM), such as a disk storage device. Alternatively, the memory 1005 may be a storage device independent of the aforementioned processor 1001.

[0024] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0025] like Figure 1 As shown, the memory 1005, which is identified as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a draft lexical prediction program.

[0026] exist Figure 1 In the device shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the user equipment; the device calls the draft lexical prediction program stored in the memory 1005 through the processor 1001 and executes the steps of the draft lexical prediction method provided in the embodiments of this application.

[0027] It's important to note that speculative decoding is a technique used to accelerate inference in large language models. In the autoregressive generation process of large language models, each generated lexical requires a complete forward propagation, resulting in significant computational overhead and high inference latency. Speculative decoding introduces a draft model with fewer parameters to quickly generate multiple consecutive candidate lexicals. These candidate lexicals are then validated in parallel by the original large model (main model), thus merging multiple sequential forward propagations into a single parallel computation, significantly improving inference efficiency.

[0028] Currently, existing speculative decoding schemes have achieved good results in single-master model scenarios. These schemes typically equip a single master model with a corresponding draft model, and train the draft model to fit the output distribution of the master model through distillation. However, in multi-low-rank adaptive inference scenarios, the actual effective weights of the base model change dynamically based on the loading of corresponding low-rank adaptation adapters according to different user requests. This means that even with the same input context, the vocabulary probability distribution of the master model's output will differ significantly after attaching different low-rank adaptation adapters. Existing speculative decoding schemes, equipped only with a single draft model, cannot simultaneously fit the dynamic probability distribution of the master model after attaching different low-rank adaptation weights.

[0029] Therefore, to address the aforementioned deficiencies, this application provides a draft lexical prediction method, apparatus, device, and storage medium. The method includes: extracting the input sequence to be predicted and a low-rank adaptation identifier carried in a received prediction request; inputting the input sequence to be predicted and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units, wherein the preset draft model includes a gating module and at least two expert modules, the gating module being used to determine a corresponding target expert module among the expert modules based on the low-rank adaptation identifier, so that the target expert module predicts the input sequence to be predicted to obtain the draft lexical units; inputting the draft lexical units into a target master model for verification to obtain the text prediction result of the input sequence to be predicted, wherein the target master model is obtained by superimposing the adapter corresponding to the low-rank adaptation identifier.

[0030] The draft lexical prediction method of this application extracts the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request, and inputs the input sequence to be predicted and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units. The preset draft model includes a gating module and at least two expert modules. The gating module is used to determine the corresponding target expert module among the expert modules according to the low-rank adaptation identifier, so that the target expert module predicts the input sequence to be predicted to obtain the draft lexical units. The draft lexical units are then input into the target main model for verification to obtain the text prediction result of the input sequence to be predicted. Since this application can determine the corresponding target expert module among the expert modules according to the low-rank adaptation identifier through the gating module, so that the target expert module predicts the input sequence to be predicted to obtain the draft lexical units, it can fit the dynamic probability distribution of the main model after attaching different low-rank adaptation weights.

[0031] For ease of understanding, the following is combined with Figures 2 to 8 The draft lexical prediction method provided in the embodiments of this application will be described in detail.

[0032] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the lexical prediction method for the draft of this application. The first embodiment of the lexical prediction method for the draft of this application is presented as follows: Figure 2 As shown, in this embodiment, the method is applied to the aforementioned device, and the specific method includes: Step S10: Extract the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request.

[0033] It is understood that the method in this embodiment can be applied to the above-mentioned draft lexical prediction device (hereinafter referred to as the device). The device can be any device with data processing, program running and data acquisition functions, such as a computer, server, etc., and there is no limitation in this embodiment.

[0034] It should be noted that the aforementioned prediction request can be a data packet or instruction received by the draft lexical prediction device from an external source to trigger a text prediction task, such as a call request containing the text to be processed sent by the user through a client.

[0035] It should also be noted that the above-mentioned input sequence to be predicted can be a piece of text or sequence data input by the user that needs to be predicted, such as the text fragment corresponding to the user's question "How is the weather today?"

[0036] It is understandable that the aforementioned low-rank adaptation identifier can be identification information used to indicate the task type or adapter selection criteria corresponding to the current prediction request, such as "low-rank adaptation_math" indicating a mathematical task.

[0037] In its implementation, the device first receives a prediction request from a client or other external system. This prediction request carries key information required for the subsequent prediction process. The device then parses the received prediction request, extracting two core elements: one is the input sequence to be predicted, i.e., the text content actually submitted by the user that the model needs to predict; the other is a low-rank adaptation identifier, used to inform the device of the task type or specific adapter selection corresponding to the current request. After completing the above extraction operations, the device obtains the input data for subsequent prediction processing and the identification information for selecting an expert model.

[0038] Step S20: Input the input sequence to be predicted and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft words. The preset draft model includes a gating module and at least two expert modules. The gating module is used to determine the corresponding target expert module in each of the expert modules according to the low-rank adaptation identifier, so that the target expert module can predict the input sequence to be predicted to obtain the draft words.

[0039] It should be noted that the aforementioned pre-set draft model can be a pre-trained, lightweight model used to quickly generate draft tokens. It contains a gating module and multiple expert modules, such as a multi-token prediction model based on an expert hybrid architecture. Specifically, it can use the hidden layer state and probability distribution obtained from the forward propagation of the main model as soft labels, shift the training corpus to the right by multiple positions to obtain the shift sequence, and concatenate the hidden layer state of the main model with the first shift sequence as the first step input. Subsequent steps concatenate the hidden layer state output by the previous draft model with the current shift sequence as input, and update the draft model parameters by calculating the weighted loss, thereby training the model.

[0040] It should also be noted that the aforementioned gating module can be the component in the aforementioned preset draft model responsible for selecting based on the aforementioned low-rank adaptation identifier, used to determine which expert module should handle the current request, such as a router that performs dynamic route allocation based on the low-rank adaptation identifier.

[0041] It should be understood that the aforementioned expert modules can be prediction sub-models that have been independently trained in the pre-set draft model and correspond to different task types. Each expert module is good at handling prediction tasks in a specific domain or style. Specifically, it can be a multi-word prediction (MTP) expert module.

[0042] It is also important to understand that the aforementioned target expert module can be the expert module selected by the gating module from multiple expert modules based on the low-rank adaptation identifier, specifically designed to handle the current prediction request.

[0043] It should be noted that the above draft lexical units can be candidate prediction results output by the target expert module for subsequent verification by the main model. For example, if the input sequence is "I want to go", the draft model can generate the draft lexical unit sequence "public", "park", "play".

[0044] In its implementation, after extracting the input sequence to be predicted and the low-rank adaptation identifier, the device passes these two elements as input to a pre-defined draft model. This pre-defined draft model is internally configured with a gating module and at least two expert modules. When the device inputs the low-rank adaptation identifier into the pre-defined draft model, the gating module parses the identifier and, based on its content, determines a target expert module from among multiple expert modules that matches the current task type. Subsequently, the device passes the input sequence to be predicted to the target expert module. The target expert module performs prediction processing on the input sequence to be predicted, generating corresponding draft terms. Through this process, the device obtains the draft terms output by the pre-defined draft model for subsequent validation by the main model.

[0045] Step S30: Input the draft lexical units into the target main model for verification to obtain the text prediction result of the input sequence to be predicted. The target main model is obtained by superimposing the adapter corresponding to the low-rank adaptation identifier.

[0046] It should be noted that the aforementioned target master model can be a large model used to perform text prediction tasks, formed by superimposing adapters corresponding to low-rank adaptation identifiers. For example, it could be a base model equipped with a specific low-rank adaptation adapter. The base model typically employs autoregressive pre-training, pre-training on a massive general corpus using the next word prediction task to obtain basic language representation capabilities. The low-rank adaptation adapter, on the other hand, is fine-tuned using a small amount of labeled data based on the base model for a specific task or domain (such as code generation, poetry creation, or dialogue systems). During training, all parameters of the base model are frozen, and only the low-rank adaptation matrix is ​​updated, thus achieving model adaptation to a specific task with a minimal number of parameters. In this scheme, the target master model is the one that dynamically loads the corresponding pre-trained adapter based on the low-rank adaptation identifier in the request during inference and superimposes it with the base model, thereby obtaining a master model that combines general knowledge with specific task capabilities.

[0047] It should also be noted that the adapter mentioned above can be a low-rank adaptation module, used to perform low-rank adjustments on the weights of the base model to adapt the model to a specific task, such as a low-rank adapter trained for a poetry generation task.

[0048] It is understandable that the above-mentioned adapter stacking can be a process of merging the weights of the adapter with the weights of the base model. For example, the parameter matrix of the low-rank adaptive adapter can be stacked onto the corresponding layer of the base model to form a task-specific model version.

[0049] In its implementation, after obtaining the draft words output by the preset draft model, the device then needs to validate these draft words using the target master model. First, the device determines the adapter corresponding to the low-rank adaptation identifier extracted in the previous step. Then, the device superimposes the determined adapter onto the base model to form the target master model specifically for this prediction task. Next, the device inputs the draft words generated by the preset draft model into the target master model. The target master model validates the draft words, and through this process, the device can obtain the prediction result corresponding to the input sequence to be predicted.

[0050] To facilitate understanding, the following explanation uses examples, but does not impose specific limitations on this embodiment. For instance, in the previous step, the device obtains the draft words "spring," "wind," "blow," and "green" output by the preset draft model. Next, based on the low-rank adaptation identifier "LoRA_poetry," the device finds the corresponding poetry adapter in the system and superimposes this poetry adapter onto the base model to form the target main model. Subsequently, the device inputs this set of draft words "spring," "wind," "blow," and "green" into the target main model. The target main model performs parallel verification on these draft words, finding that the first three words "spring," "wind," and "blow" are correctly predicted, while "green" is incorrect. Based on the verification results, the target main model generates the final correct sequence "spring breeze blows," which serves as the text prediction result for the input sequence "Please help me write a poem about spring."

[0051] Furthermore, in order to obtain the target main model and the hidden layer state information of the main model, and to improve the accuracy of the preset draft model in predicting draft words, in this embodiment, the step of inputting the input sequence to be predicted and the low-rank adaptation identifier into the preset draft model for prediction to obtain draft words includes: Step S21: Determine the corresponding adapter based on the low-rank adaptation identifier, and superimpose the adapter onto the base large model to obtain the target main model.

[0052] It should be noted that the aforementioned base model can be a pre-trained large language model with general text generation and understanding capabilities, such as GPT and other models.

[0053] It should also be noted that the aforementioned target master model can be a large model specifically designed for the current prediction task, formed by superimposing a base model with a specific adapter, such as a mathematical model formed by superimposing a mathematical adapter on a base model.

[0054] In its implementation, the device first searches for the corresponding adapter in a pre-defined adapter library based on the extracted low-rank adaptation identifier. The device determines the specific adapter for the current prediction request through identifier matching. Next, the device loads the determined adapter into the base model and superimposes the adapter's weights with the weights of the base model. Through this superposition operation, the base model gains specific optimization capabilities for the current task while maintaining its original general capabilities. After this process, the device obtains a target master model specifically for the current prediction request.

[0055] Step S22: Input the input sequence to be predicted into the target master model for forward propagation to generate the hidden layer state information of the master model corresponding to the input sequence to be predicted.

[0056] It should be noted that the aforementioned forward propagation can be the process by which a neural network model processes input data, calculating layer by layer from the input layer through each hidden layer until the output layer. For example, after inputting a text sequence into a large model, the model calculates layer by layer to obtain intermediate representations and the final output.

[0057] It should also be noted that the hidden layer state information of the main model mentioned above can be the intermediate feature representation output from one or more layers inside the target main model after the forward propagation is completed. This information contains the model's high-dimensional semantic understanding of the input sequence. For example, after the target main model receives the input "How is the weather today?", it outputs a high-dimensional feature vector after processing through multiple transformer layers.

[0058] In its implementation, after obtaining the target master model, the device then passes the input sequence to be predicted as input data to the target master model. The device triggers the target master model to perform a forward propagation operation. During the forward propagation of the target master model, the input sequence to be predicted sequentially passes through components such as the model's embedding layer, multiple transformer hidden layers, and vocabulary projection layer. The device extracts processed feature information from the internal layers of the target master model; this feature information is the hidden layer state information of the master model corresponding to the input sequence to be predicted. Subsequently, the device uses this hidden layer state information as intermediate data for subsequent input into a preset draft model to generate draft lexical units.

[0059] Step S23: Input the input sequence to be predicted, the hidden layer state information of the main model corresponding to the input sequence to be predicted, and the low-rank adaptation identifier into the preset draft model for prediction to obtain draft lexical units.

[0060] In its implementation, after obtaining the hidden layer state information of the master model corresponding to the input sequence to be predicted in the previous step, the device then passes this hidden layer state information, along with the input sequence to be predicted and the low-rank adaptation identifier, as input data to a preset draft model. The device simultaneously inputs these three elements into the preset draft model. Upon receiving the input, the preset draft model first parses the low-rank adaptation identifier through a gating module to determine the target expert module that matches the current request. Subsequently, the device uses the target expert module to jointly process the input sequence to be predicted and the hidden layer state information of the master model, predicting multiple candidate draft terms. Finally, the device obtains these draft terms from the preset draft model as candidate prediction results subsequently validated by the target master model.

[0061] Furthermore, in order to more accurately extract features from the input sequence and hidden layer state information, and to improve the efficiency and accuracy of draft lexical generation, refer to Figure 3 , Figure 3This is a module connection diagram of the preset draft model in the first embodiment of the draft lexical prediction method of this application, as shown below. Figure 3 As shown, in this embodiment, the preset draft model also includes: an embedding module, a projection module, and a decoding sampling module; Furthermore, the step of inputting the input sequence to be predicted, the hidden layer state information of the main model corresponding to the input sequence to be predicted, and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units includes: Step S231: Extract features from the input sequence to be predicted and the hidden layer state information of the main model corresponding to the input sequence to be predicted through the embedding module to obtain fused features.

[0062] It should be noted that the aforementioned embedding module can be a component in a pre-defined draft model used to convert the input sequence and hidden layer state information into a continuous vector representation, such as an embedding layer that maps discrete words to high-dimensional feature vectors.

[0063] It should also be noted that the above feature extraction can be a process of encoding and fusing the input text sequence and feature vectors, such as concatenating or merging the text information of the input sequence to be predicted with the hidden layer state information of the main model in the feature dimension.

[0064] It is understandable that the aforementioned fusion feature can be a unified feature representation that integrates the information of the input sequence and the hidden layer state information obtained by the embedding module after extracting features from the input sequence and the hidden layer state information. For example, it can be a combined feature vector formed by concatenating the embedding vector of the text sequence with the hidden layer state vector of the main model.

[0065] In its implementation, after obtaining the input sequence to be predicted and the hidden layer state information of the main model, the device passes these two as input data to the embedding module in the pre-defined draft model. The embedding module first encodes the input sequence to be predicted, converting it into a sequence feature vector; simultaneously, it also performs feature mapping on the hidden layer state information of the main model. Subsequently, the embedding module fuses these two feature information sets to obtain a unified fused feature representation. Through this method, the device obtains a comprehensive feature that integrates the original input information and the deep semantic information of the main model. This comprehensive feature will be used for subsequent processing by the expert module.

[0066] To facilitate understanding, the following explanation uses examples, but does not impose specific limitations on this embodiment. For instance, in the previous step, the device obtains the input sequence to be predicted as "Calculate what 3 plus 5 equals," and the hidden layer state information of the main model is a high-dimensional feature vector used to characterize the semantics of the main model's output response. The device inputs both of these into the embedding module. The embedding module first converts the text "Calculate what 3 plus 5 equals" into corresponding word embedding vectors, and then concatenates these embedding vectors with the hidden layer state vector of the main model in terms of feature dimensions, forming a higher-dimensional fusion feature vector. This fusion feature vector characterizes the joint context of the user input and the semantics of the main model's output response. The device then obtains this fusion feature vector for subsequent processing by the target expert module.

[0067] Step S232: Determine the target expert module corresponding to the low-rank adaptation identifier through the gating module.

[0068] In its implementation, after obtaining the fusion features, the device inputs a low-rank adaptation identifier into a gating module within a pre-defined draft model. The gating module parses the received low-rank adaptation identifier to identify the task type or adapter type indicated by the identifier. Subsequently, the gating module searches among all available expert modules to determine the expert module corresponding to the low-rank adaptation identifier as the target expert module. After the device completes this selection through the gating module, it can then pass the subsequent fusion features to the target expert module for processing.

[0069] Step S233: The target expert module processes the fusion feature to obtain the hidden layer state information of the expert module corresponding to the fusion feature.

[0070] It should be noted that the hidden layer state information of the aforementioned expert module can be an intermediate feature representation output by the target expert module after receiving the fused features and processing them through the internal network layer. This feature information contains the target expert module's further encoding and understanding of the fused features, such as a high-dimensional feature vector output by the mathematical expert module after processing the fused features, used to predict mathematical answers.

[0071] In its implementation, after determining the target expert module through the gating module, the device inputs the fused features obtained in the previous step into the target expert module. The target expert module performs further forward propagation processing on the fused features, transforming them into a deeper feature representation through its internal linear mapping layer, attention hidden layer, and other components. The device then obtains the processed feature information from the output of the target expert module; this feature information is the hidden layer state information of the expert module corresponding to the fused features. Subsequently, the device uses this hidden layer state information as an intermediate result for subsequent projection module processing.

[0072] Step S234: The hidden layer state information of the expert module corresponding to the fusion feature is mapped to a vocabulary probability distribution through the projection module.

[0073] It should be noted that the aforementioned projection module can be a component in a pre-defined draft model used to convert hidden layer state information into a word probability distribution, such as a word projection layer that shares weights with the main model; the aforementioned word probability distribution can be the predicted probability value for each word in the word list output by the projection module, used to indicate the probability of each word being the next prediction result, such as a vector containing all words in the word list and their corresponding probabilities.

[0074] In its implementation, after obtaining the hidden layer state information from the expert module, the device inputs this information into the projection module of the pre-defined draft model. The projection module performs a linear transformation on the received hidden layer state information, mapping it to the vocabulary dimension. This mapping operation converts the high-dimensional hidden layer features into predicted probability values ​​for each word in the vocabulary. After this processing, the device obtains a vocabulary probability distribution, which contains the probability of each word in the vocabulary being the next prediction result. The device then passes this vocabulary probability distribution to the decoding sampling module for subsequent draft word sampling.

[0075] Step S235: Obtain draft lexical units by decoding and sampling according to the probability distribution of the vocabulary through the decoding and sampling module.

[0076] It should be noted that the above-mentioned decoding sampling module can be a component in the preset draft model used to select specific words based on the probability distribution, such as a decoder that extracts one or more words from the vocabulary probability distribution based on a sampling strategy; the above-mentioned decoding sampling can be a process of selecting one or more words from the vocabulary probability distribution according to their probability magnitude as the prediction result, such as selecting the word with the highest probability as the output, or performing random sampling according to the probability ratio.

[0077] In its implementation, after obtaining the vocabulary probability distribution output by the projection module, the device inputs this probability distribution into the decoding sampling module of the preset draft model. The decoding sampling module selects one or more words as output results based on the probability values ​​corresponding to each word in the vocabulary probability distribution, according to a preset sampling strategy. For example, the decoding sampling module can select the word with the highest probability value as the current prediction result, or it can use random sampling to extract words from the probability distribution. Through this process, the device obtains one or more draft words as prediction results. The device then passes the obtained draft words to the target main model for subsequent verification processing.

[0078] refer to Figure 4 , Figure 4 This is a flowchart of the overall workflow of the first embodiment of the lexical prediction method in the draft of this application, as follows: Figure 4 As shown, in this embodiment, the prediction request first enters the main model module. After forward propagation, the base model (i.e., the base large model) dynamically loads the corresponding LoRA adapter according to the identifier carried in the request, generates context information (i.e., the input sequence to be predicted), hidden layer information (i.e., the hidden layer state information of the main model), and LoRA identifier information (i.e., the low-rank adaptation identifier), and transmits it to the multi-term prediction-expert hybrid (MTP-MOE) architecture module (i.e., the preset draft model). This information first undergoes unified feature extraction through the shared embedding layer (i.e., the embedding module), and then the gate controller (i.e., the gate control module) performs dynamic routing according to the LoRA identifier information, accurately activating the corresponding dedicated MTP expert (i.e., the target expert module) in the MTP hybrid expert layer. After the activated MTP expert processes the features, it passes the result to the shared vocabulary projection layer (i.e., the projection module), and then through the decoding sampling layer (i.e., the decoding sampling module) to generate continuous draft tokens (i.e., draft words). Finally, the draft words are returned to the main model module.

[0079] In this embodiment, the gating module determines the corresponding target expert module in each of the expert modules based on the low-rank adaptation identifier, so that the target expert module can predict the input sequence to be predicted and obtain the draft lexical units, thereby fitting the dynamic probability distribution of the main model after attaching different low-rank adaptation weights.

[0080] Reference Figure 5 , Figure 5 This is a flowchart illustrating the second embodiment of the draft lexical prediction method of this application. Based on the first embodiment described above, a second embodiment of the draft lexical prediction method of this application is proposed.

[0081] To obtain the aforementioned pre-defined draft model and ensure that the draft lexical units maintain a high acceptance rate even when different low-rank adaptation weights are attached to the main model, in this embodiment, before the step of extracting the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request, the following steps are included: Step S01: Obtain the training corpus sequence and input the training corpus sequence into the target main model for forward propagation to generate the hidden layer state information of the main model and the first probability distribution corresponding to the training corpus sequence.

[0082] It should be noted that the training corpus sequence mentioned above can be a text data sequence used to train a pre-defined draft model, such as a complete text sequence used to train a specific task expert, specifically T={t0, t1, t2, ..., tn}. Furthermore, the training corpus can be obtained as follows: each MTP expert is trained independently using specific task corpus matching its mapped LoRA adapter, ensuring data isolation and feature decoupling among experts. The system extracts the corresponding prompts based on the type of LoRA adapter currently mounted on the main model and inputs them into the main model. The system uses the response sequence generated by the main model's forward inference as the target data (i.e., self-generated corpus). The prompts and generated responses are concatenated to form the complete training sequence for that specific MTP expert. The aforementioned first probability distribution can be the predicted probability distribution of the next word corresponding to one or more positions in the training corpus sequence, output by the target main model after completing forward propagation. Specifically, it can be denoted as P_target. This probability distribution can be decoded into the prediction sequence t1, t2, t3, ..., tn+1 through the argmax operation. In practical applications, the first probability distribution can be a prediction probability distribution for a single position, such as the probability prediction result of the next word at a specific position in the training corpus sequence; or it can be a prediction probability distribution for multiple positions, such as the probability prediction result of the next word corresponding to each of multiple consecutive positions in the training corpus sequence. That is to say, when there are prediction probability distributions for multiple positions, each position has a corresponding first probability distribution output by the aforementioned target main model.

[0083] In its implementation, the device first acquires a pre-prepared training corpus sequence. This training corpus sequence is a complete text data set used to train a pre-defined draft model, containing all information from the first lexical unit to the last. Next, the device inputs the acquired training corpus sequence into the target master model for forward propagation. The target master model performs layer-by-layer computation on the training corpus sequence, processing it through components such as embedding layers and multiple transformer hidden layers, and outputs the hidden layer state information corresponding to the training corpus sequence. Simultaneously, the target master model also outputs predicted probability distributions for the next lexical unit corresponding to each position in the training corpus sequence through a vocabulary projection layer. These predicted probability distributions are called the first probability distributions. The first probability distribution can be a prediction result for a single position, such as the probability prediction result for the next lexical unit at a specific position in the training corpus sequence; or it can be prediction results for multiple positions, such as the probability prediction results for the next lexical unit corresponding to each of several consecutive positions in the training corpus sequence. In the case of multiple position prediction results, each position has its corresponding first probability distribution. The aforementioned device then obtains the hidden layer state information of the main model and the first probability distribution (or multiple first probability distributions) for subsequent draft model training steps.

[0084] Step S02: Input the hidden layer state of the main model corresponding to the training corpus sequence into the initial draft model to obtain the second probability distribution.

[0085] It should be noted that the aforementioned second probability distribution can be the predicted probability distribution output by the initial draft model after receiving the hidden layer state information of the main model. More specifically, it can be denoted as P_predict_1. This probability distribution can be compared with the first probability distribution to calculate the training loss. More specifically, the loss function can be defined as: Loss_1=Cross_Entropy(P_predict,P_target); Where Loss_1 is the loss, Cross_Entropy is the cross-entropy function used to calculate the difference between the predicted probability distribution and the target probability distribution of the main model, P_predict represents the second probability distribution, and P_target represents the first probability distribution.

[0086] In practical applications, the second probability distribution can be a prediction probability distribution for a single position, such as the probability prediction result for the next word at a specific position in the training corpus sequence; or it can be a prediction probability distribution for multiple positions, such as the probability prediction results for the next word corresponding to each of several consecutive positions in the training corpus sequence. When there are prediction probability distributions for multiple positions, each position has a corresponding second probability distribution output by the initial draft model.

[0087] In its implementation, after obtaining the hidden layer state information of the main model, the device passes this state information as input data to the initial draft model. The initial draft model includes a gating module and multiple expert modules. During the training phase, since the training corpus sequence is constructed for a specific low-rank adaptation task, the gating module, upon receiving the hidden layer state of the main model corresponding to the training corpus, automatically activates the target expert module corresponding to that task according to a preset task mapping relationship, without needing to explicitly parse the low-rank adaptation identifier from the input information. The activated target expert module performs forward propagation processing on the hidden layer state information of the main model. After layer-by-layer calculation by its internal components, it outputs a prediction probability distribution corresponding to one or more positions to be predicted in the training corpus sequence. For a single position to be predicted, a second probability distribution is output; for multiple positions to be predicted, separate second probability distributions are output for each position. The device then obtains this second probability distribution, compares it with the first probability distribution, and calculates the training loss.

[0088] Step S03: Determine the loss based on the first probability distribution and the second probability distribution, and update the initial draft model based on the loss to obtain a preset draft model.

[0089] It should be noted that the aforementioned loss can be a numerical indicator used to measure the difference between the first probability distribution and the second probability distribution, such as the cross-entropy loss value. The smaller the value, the closer the prediction result of the initial draft model is to the prediction result of the target main model. The aforementioned preset draft model can be a draft model that has the ability to predict draft words after being updated through the above training process, such as a draft model that can be used to perform text prediction tasks after training.

[0090] In its implementation, after obtaining the first and second probability distributions, the device calculates the difference between them. When each probability distribution corresponds to a single position to be predicted, the device directly calculates the difference between the first and second probability distributions at that position, obtaining a loss value. When each probability distribution corresponds to multiple positions to be predicted, the device calculates the difference between the first and second probability distributions for each position, obtaining a loss value for each position. Subsequently, based on the calculated loss values ​​(or the sum of the loss values ​​for multiple positions), the device uses the backpropagation algorithm to calculate the gradients of each parameter in the initial draft model and updates the parameters of the initial draft model according to the gradients, so that the updated model can produce results closer to the first probability distribution in the next prediction. The device repeats the above training process until the model converges, ultimately obtaining a pre-trained draft model that can be used for subsequent text prediction tasks.

[0091] Furthermore, in order to achieve continuous prediction of multiple positions to be predicted in the training corpus sequence to support the multi-word output capability of the draft model, in this embodiment, the step of inputting the training corpus sequence into the target main model for forward propagation to generate the hidden layer state information of the main model corresponding to the training corpus sequence and the first probability distribution includes: S011: Input the training corpus sequence into the target main model for forward propagation to generate the hidden layer state information of the main model corresponding to the training corpus sequence and the first probability distribution corresponding to each position to be predicted in the training corpus sequence.

[0092] The aforementioned positions to be predicted can be all positions in the training corpus sequence that the model needs to predict the next word. For example, in the training corpus sequence "3 plus 5 equals what, equals 8", the words "3", "plus", "5", "equal to", "at", "some", ",", "equal to", and "at" each correspond to a position to be predicted, and each position to be predicted needs to predict the next word. The aforementioned first probability distribution refers to an independent probability distribution output by the target main model for each position to be predicted in the training corpus sequence, which is used to predict the next word after that position.

[0093] In its implementation, after acquiring the training corpus sequence, the device inputs the training corpus sequence into the target main model. The target main model performs a complete forward propagation process on the training corpus sequence, performing layer-by-layer calculations through components such as the embedding layer and multiple transformer hidden layers, and outputs the main model hidden layer state information corresponding to the training corpus sequence. Simultaneously, the target main model maps the hidden layer state information through a vocabulary projection layer, and for each position to be predicted in the training corpus sequence, outputs the prediction probability distribution of the next word corresponding to that position. Each position to be predicted corresponds to an independent first probability distribution. The device then obtains the main model hidden layer state information and the first probability distributions corresponding to each position to be predicted, for use in subsequent draft model training steps.

[0094] The step of inputting the hidden layer state of the main model corresponding to the training corpus sequence into the initial draft model to obtain the second probability distribution includes: Step S021: Gradually shift the training corpus sequence to the right to obtain at least two shifted sequences.

[0095] It should be noted that the above-mentioned offset sequence can be a sequence obtained by shifting the training corpus sequence to the right by a certain number of positions. For example, shifting the sequence "3 plus 5 equals what, equals 8" to the right by 1 position will result in "plus 5 equals what, equals 8". The above-mentioned stepwise offset can be a process of shifting the training corpus sequence to the right multiple times in succession, shifting one position each time to obtain multiple offset sequences. For example, shifting to the right by 1 position, 2 positions, 3 positions, etc., in sequence will yield the corresponding offset sequences.

[0096] In its implementation, after obtaining the training corpus sequence, the device performs a progressive offset operation on the sequence. Specifically, it shifts the training corpus sequence one position to the right to obtain an offset sequence; then shifts it another position to the right to obtain a second offset sequence; and so on, until at least two offset sequences are obtained. Through this progressive offset operation, the device obtains multiple offset sequences, which will be used as prediction inputs for different subsequent prediction positions.

[0097] Step S022: Based on the hidden layer state of the main model corresponding to the training corpus sequence and the first offset sequence, obtain the second probability distribution of the first position to be predicted.

[0098] It should be noted that the first offset sequence mentioned above can be the sequence obtained by shifting the training corpus sequence one position to the right, such as "add 5 equals what, equals 8"; the first position to be predicted mentioned above can be the position of the first word in the training corpus sequence that needs to be predicted next, such as the position of "3".

[0099] In its implementation, after obtaining the hidden layer state of the main model corresponding to the training corpus sequence and the first offset sequence, the device fuses the hidden layer state of the main model with the first offset sequence. Specifically, the device concatenates or combines the word embedding representations of the hidden layer state of the main model and the first offset sequence to obtain a fused feature. This fused feature is then input into the target expert module in the initial draft model. The target expert module performs forward propagation processing on the fused feature, passing through its internal linear mapping layer, attention hidden layer, and other components, and outputs a predicted probability distribution, which is the second probability distribution corresponding to the first position to be predicted. The device then obtains this second probability distribution for subsequent comparison with the first probability distribution output by the target main model.

[0100] Step S023: Based on the hidden layer state information of the expert module corresponding to the previous position of the non-first position to be predicted and the offset sequence corresponding to the non-first position to be predicted, obtain the second probability distribution of the non-first position to be predicted.

[0101] It should be noted that the aforementioned non-first prediction position can be any other prediction position in the training corpus sequence besides the first prediction position. For example, in the sequence "3 plus 5 equals what, equals 8", the positions corresponding to "plus", "5", "equals", "at", "some", ",", "equals", and "at" are all non-first prediction positions. The hidden layer state information of the expert module corresponding to the aforementioned previous position can be the intermediate feature information output by the target expert module when predicting the previous prediction position. For example, the hidden layer state output when predicting the word after "3" can be used to predict the next position. The offset sequence corresponding to the aforementioned non-first prediction position can be the offset sequence corresponding to the current prediction position. For example, when predicting the word after "plus", the corresponding offset sequence is "5 equals what, equals 8".

[0102] In its implementation, after obtaining the second probability distribution corresponding to the first position to be predicted, the device then continues to predict subsequent positions that are not the first position to be predicted. For each non-first position to be predicted, the device first obtains the hidden layer state information of the expert module corresponding to the previous position. Specifically, this hidden layer state information is the intermediate feature representation output by the target expert module after processing by its internal components when predicting the previous position. Next, the device obtains the offset sequence corresponding to the current non-first position to be predicted. Subsequently, the device fuses the hidden layer state information of the expert module corresponding to the previous position with the offset sequence corresponding to the current non-first position to be predicted to obtain a fused feature. This fused feature combines the prediction state information from the previous step with the input information from the current step. Finally, the device inputs the fused feature to the target expert module, and after forward propagation processing, outputs the second probability distribution corresponding to the current non-first position to be predicted. The device repeats the above operation for each non-first position to be predicted, sequentially obtaining the second probability distribution corresponding to each position to be predicted.

[0103] Furthermore, in order to more reasonably measure the prediction performance of the initial draft model at multiple prediction positions and optimize the training process, in this embodiment, the step of determining the loss based on the first probability distribution and the second probability distribution, and updating the initial draft model based on the loss to obtain the preset draft model includes: Step S031: Determine the loss corresponding to each of the predicted positions based on the first probability distribution and the second probability distribution of each predicted position. It should be noted that the aforementioned loss can be a numerical indicator used to measure the difference between the first probability distribution and the second probability distribution, such as the cross-entropy loss value. The smaller the value, the closer the prediction result of the initial draft model is to the prediction result of the target main model at the corresponding prediction position.

[0104] In its implementation, after obtaining the first and second probability distributions corresponding to each position to be predicted, the device calculates the loss for each position. Specifically, the device iterates through each position to be predicted in the training corpus sequence. For each position, the first probability distribution is used as the target distribution, and the second probability distribution is used as the prediction distribution. The difference between the two is calculated to obtain a loss value. The device repeats this calculation process for each position to obtain the loss value for each position. These loss values ​​reflect the differences between the prediction results of the initial draft model at each position and the prediction results of the target main model.

[0105] Step S032: The total loss is obtained by weighted summation of the losses corresponding to each of the predicted positions according to their respective weights. The weight of the loss corresponding to each predicted position is negatively correlated with the depth of the corresponding predicted position.

[0106] It should be noted that the weights mentioned above can be numerical coefficients used to adjust the contribution of different prediction positions to the total loss. For example, the weight of the first prediction position is 1.0, the weight of the second prediction position is 0.8, the weight of the third prediction position is 0.64, and so on.

[0107] It should also be noted that the depth of the predicted position can represent its position number in the prediction sequence. For example, the depth of the first predicted position is 1, the depth of the second predicted position is 2, and the depth of the third predicted position is 3. The negative correlation mentioned above can refer to the fact that the weight decreases as the depth increases, that is, the deeper the predicted position, the smaller its loss weight. For example, the weight is 0.9 at a depth of 1, 0.81 at a depth of 2, and 0.729 at a depth of 3. Furthermore, the total loss can be understood as the comprehensive loss value obtained by weighting and summing the losses of each predicted position according to their corresponding weights. This value is used to comprehensively evaluate the prediction performance of the initial draft model at all predicted positions. Specifically, the formula for calculating the total loss function can be: ; As shown in the formula above, where: k represents the total depth of the MTP expert prediction draft tokens during training (i.e., the total number of forwards); i represents the i-th prediction (i is a positive integer and 1 <= i <= k); Cross_Entropy represents the cross-entropy function, used to calculate the difference between the predicted probability distribution (i.e., the second probability distribution) and the main model's target probability distribution (i.e., the first probability distribution); w is the weight decay parameter (w varies between 0.7 and 0.9 depending on the different MTP expert training settings). i 1 This represents w raised to the power of i-1. This weight decays exponentially with increasing prediction depth i, allowing the model to focus more on the accuracy of shallower draft tokens during training.

[0108] In its implementation, after obtaining the loss values ​​for each prediction location, the device calculates the total loss. First, it determines the weight of each prediction location based on its depth. Depth and weight are negatively correlated; the further forward the prediction location (the shallower the depth), the greater its weight, and vice versa. Next, the device multiplies the loss value of each prediction location by its corresponding weight to obtain the weighted loss for that location. Then, it sums the weighted losses of all prediction locations to obtain a comprehensive total loss. This total loss is then used to update the parameters of the initial draft model.

[0109] Step S033: Update the initial draft model according to the total loss to obtain the preset draft model.

[0110] In its implementation, after calculating the total loss, the device updates the parameters of the initial draft model based on this total loss. Specifically, the device uses the backpropagation algorithm to calculate the gradients of each trainable parameter (including parameters in the linear mapping layer and the attention hidden layer) in the initial draft model layer by layer, starting from the total loss. Subsequently, based on the calculated gradients, the device uses the gradient descent optimization algorithm to adjust the parameters of the initial draft model, so that the total loss value generated by the updated model gradually decreases in the next forward propagation. The device repeats the above training process, including forward propagation, loss calculation, backpropagation, and parameter update, until the total loss value converges to a preset threshold or reaches a preset number of training rounds. After the above training, the device obtains a trained preset draft model, which has the ability to predict draft lexical units in multi-low-rank adaptation scenarios.

[0111] Reference Figure 6 , Figure 6 This is a flowchart of the overall training process in the second embodiment of the lexical prediction method in the draft of this application, as follows: Figure 6 As shown, Figure 6Taking a two-layer prediction depth as an example, the base model first attaches a corresponding LoRA adapter, inputs the training sequence (t0-t3) to complete forward propagation, generates hidden states, and obtains the target output (t1-t4) through a shared vocabulary projection layer. The MTP expert module shares the embedding layer and vocabulary projection layer of the base model, and the parameters of these two layers are frozen throughout. Then, it performs two forward propagations: the first input is a sequence offset by 1 position (t1-t4), which is processed by the shared embedding layer to extract features and concatenated with the hidden states from the base model. It then passes through a linear projection layer and a transformer block to generate hidden states, and then through the shared vocabulary projection layer to calculate the first loss (loss-mtp1). The first loss is the loss between the probability distribution of the draft model output and the probability distribution of the target main model output in the first prediction. The second input is a sequence offset by 2 positions (t2-t5), which is processed by the shared embedding layer to extract features and concatenated with the hidden states generated in the first forward propagation. It then passes through a linear projection layer and a transformer block to generate hidden states. The Block and the shared vocabulary projection layer are used to calculate the second loss (loss-mtp2), which is the loss between the probability distribution output by the draft model and the probability distribution output by the target main model in the second prediction. Finally, loss-mtp1 and loss-mtp2 are summed with exponentially decaying weights to obtain the total loss. Backpropagation is then used to update only the linear projection layer and Transformer Block parameters of the MTP expert until the model converges.

[0112] Reference Figure 7 , Figure 7 This is a flowchart illustrating the third embodiment of the draft lexical prediction method of this application. Based on the above embodiments, a third embodiment of the draft lexical prediction method of this application is proposed. In order to efficiently verify the draft lexical units generated by the preset draft model and concatenate the verified lexical units into the current context sequence to complete the generation of the text prediction result, in this embodiment, the step of inputting the draft lexical units into the target main model for verification and obtaining the text prediction result of the input sequence to be predicted includes: Step S31: Group the draft lexical units in order to obtain several lexical units, and verify each lexical unit at the same time, with the number of lexical units in each lexical unit increasing sequentially.

[0113] It should be noted that the above-mentioned token groups can be multiple subsequences obtained by splitting the draft token sequence in order. Each token group contains a continuous prefix in the draft token sequence. For example, if the draft token sequence is "spring", "wind", "blows", "greens", then the token groups can include "spring", "spring wind", "spring wind blows", "spring wind blows greens"; the above verification can be a process in which the target main model performs forward propagation verification on the token group to determine whether each token in the token group is consistent with the prediction result of the target main model. For example, the target main model verifies whether both "spring" and "wind" in the group "spring wind" are correctly predicted.

[0114] In a specific implementation, after the above device obtains the draft token sequence generated by the preset draft model, it then performs grouping processing on the draft token sequence. Specifically, the above device constructs multiple token groups in sequence according to the arrangement order of the draft tokens: the first token group contains the first draft token; the second token group contains the first and second draft tokens; the third token group contains the first, second, and third draft tokens; and so on until the last token group contains all the draft tokens. By the above method, the above device obtains several token groups, and the number of tokens in each token group increases sequentially from front to back. Subsequently, the above device inputs these token groups into the target main model simultaneously for parallel verification. The target main model performs forward propagation on each token group respectively to verify whether the draft tokens in each token group are consistent with its own prediction result.

[0115] Step S32: In the case where there is a token group with verification failure among the token groups, determine the target token group with the smallest number of token groups and verification failure, and splice the previous token group of the target token group with the to-be-predicted input sequence to obtain a new to-be-predicted sequence.

[0116] It should be noted that the above-mentioned token group with verification failure can be a token group determined by the target main model to have at least one incorrect token during the parallel verification process. For example, the token group "spring wind blows" fails the verification at the third position "blows"; the above target token group can be the one with the smallest number of tokens among all the token groups with verification failure. For example, if both "spring wind blows" and "spring wind blows greens" fail the verification, then "spring wind blows" is the target token group; the above previous token group can be the token group with one less token than the target token group before the target token group. For example, if the target token group is "spring wind blows", then the previous token group is "spring wind"; the above new to-be-predicted sequence can be a new input sequence obtained by splicing the previous token group with the original to-be-predicted input sequence and is used as the input for the next round of speculative decoding.

[0117] In its implementation, after completing parallel validation of all tuples, the device first determines whether any tuples failed validation. If such tuples exist, the device selects the one with the fewest tuples from the list and designates it as the target tuple. Next, the device finds the preceding tuple of the target tuple—that is, the tuple with one fewer tuple than the target tuple and which passed all validations. Then, the device concatenates this preceding tuple with the current input sequence to be predicted, forming a new sequence to be predicted. This new sequence contains the original input sequence and the longest draft prefix confirmed by the main model, used for further predictions. The device then returns to execute the previous prediction steps to continue generating subsequent draft tuples.

[0118] Step S33: Return to the step of inputting the input sequence to be predicted and the low-rank adaptation identifier into the preset draft model for prediction to obtain draft words, until there are no failed word groups in each word group, and then concatenate the latest draft word group with the latest input sequence to be predicted as the text prediction result.

[0119] It should be noted that the latest draft lexicon can be the draft lexicon sequence generated by the preset draft model in the last prediction and passed the target main model verification; the latest input sequence to be predicted can be the input sequence to be predicted formed after the previous round of concatenation in the last prediction; the text prediction result can be the final output, complete text prediction sequence, such as the complete response content of the user's request.

[0120] In its implementation, after generating and validating draft lexical units in one round, if there are any failed validation lexical units, the device, according to the processing logic of the previous step, concatenates the preceding lexical unit of the smallest failed validation lexical unit with the current input sequence to be predicted, obtaining a new sequence to be predicted. Then, the device returns to the previous step of "inputting the input sequence to be predicted and the low-rank adaptation identifier into the preset draft model for prediction to obtain draft lexical units," thus starting a new round of cyclic prediction. During each round of prediction, the device repeats the steps of draft generation, parallel validation, and failure handling.

[0121] The device continues to execute the aforementioned cyclic process until, in a certain prediction, all tuples (from shortest to longest) pass verification, meaning there are no tuples that fail verification. At this point, the device concatenates all draft tuples generated in the current prediction (i.e., the longest tuple) with the current input sequence to be predicted, forming the final complete sequence. The device then outputs this complete sequence as the text prediction result of the input sequence to be predicted.

[0122] In addition, refer to Figure 8 , Figure 8 This is a structural block diagram of the lexical prediction device in the draft of this application. Figure 8 As shown in the embodiments of this application, a draft lexical prediction device is also proposed, which includes: The information extraction module 801 is used to extract the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request. The lexical acquisition module 802 is used to input the input sequence to be predicted and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units. The preset draft model includes a gating module and at least two expert modules. The gating module is used to determine the corresponding target expert module in each of the expert modules according to the low-rank adaptation identifier, so that the target expert module can predict the input sequence to be predicted to obtain the draft lexical units. The lexical verification module 803 is used to input the draft lexical into the target main model for verification, and obtain the text prediction result of the input sequence to be predicted. The target main model is obtained by superimposing the adapter corresponding to the low-rank adaptation identifier.

[0123] Furthermore, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the draft lexical prediction method described above.

[0124] In this embodiment, the device uses a gating module to determine the corresponding target expert module among the expert modules based on the low-rank adaptation identifier, so that the target expert module can predict the input sequence to be predicted and obtain the draft lexical units, thereby fitting the dynamic probability distribution of the main model after attaching different low-rank adaptation weights.

[0125] Other embodiments or specific implementations of the draft lexical prediction device described in this example can be found in the above-described method embodiments, and will not be repeated here.

[0126] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0127] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0129] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for predicting lexical terms in drafts, characterized in that, The method includes: Extract the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request; The input sequence to be predicted and the low-rank adaptation identifier are input into a preset draft model for prediction to obtain draft words. The preset draft model includes a gating module and at least two expert modules. The gating module is used to determine the corresponding target expert module in each of the expert modules according to the low-rank adaptation identifier, so that the target expert module can predict the input sequence to be predicted to obtain the draft words. The draft lexical units are input into the target main model for verification to obtain the text prediction result of the input sequence to be predicted. The target main model is obtained by superimposing the adapter corresponding to the low-rank adaptation identifier.

2. The method as described in claim 1, characterized in that, The step of inputting the input sequence to be predicted and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units includes: The corresponding adapter is determined based on the low-rank adaptation identifier, and the adapter is superimposed on the base model to obtain the target main model; The input sequence to be predicted is input into the target main model for forward propagation to generate the hidden layer state information of the main model corresponding to the input sequence to be predicted. The input sequence to be predicted, the hidden layer state information of the main model corresponding to the input sequence to be predicted, and the low-rank adaptation identifier are input into the preset draft model for prediction to obtain draft lexical units.

3. The method as described in claim 2, characterized in that, The preset draft model also includes: an embedding module, a projection module, and a decoding sampling module; The step of inputting the input sequence to be predicted, the hidden layer state information of the main model corresponding to the input sequence to be predicted, and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units includes: The embedding module extracts features from the input sequence to be predicted and the hidden layer state information of the main model corresponding to the input sequence to be predicted to obtain fused features. The gating module determines the target expert module corresponding to the low-rank adaptation identifier. The target expert module processes the fused features to obtain the hidden layer state information of the expert module corresponding to the fused features; The projection module maps the hidden layer state information of the expert module corresponding to the fusion feature into a vocabulary probability distribution. The decoding and sampling module obtains draft lexical units by decoding and sampling according to the probability distribution of the vocabulary.

4. The method as described in claim 2, characterized in that, Before the step of extracting the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request, the following steps are included: Obtain the training corpus sequence and input the training corpus sequence into the target main model for forward propagation to generate the hidden layer state information of the main model and the first probability distribution corresponding to the training corpus sequence; Input the hidden layer state of the main model corresponding to the training corpus sequence into the initial draft model to obtain the second probability distribution; The loss is determined based on the first probability distribution and the second probability distribution, and the initial draft model is updated based on the loss to obtain a preset draft model.

5. The method as described in claim 4, characterized in that, The step of inputting the training corpus sequence into the target main model for forward propagation to generate the hidden layer state information of the main model corresponding to the training corpus sequence and the first probability distribution includes: The training corpus sequence is input into the target main model for forward propagation to generate the hidden layer state information of the main model corresponding to the training corpus sequence and the first probability distribution corresponding to each position to be predicted in the training corpus sequence. The step of inputting the hidden layer state of the main model corresponding to the training corpus sequence into the initial draft model to obtain the second probability distribution includes: The training corpus sequence is gradually shifted to the right to obtain at least two shifted sequences; Based on the hidden layer state of the main model corresponding to the training corpus sequence and the first offset sequence, the second probability distribution of the first position to be predicted is obtained; The second probability distribution of the non-first prediction position is obtained by combining the hidden layer state information of the expert module corresponding to the previous position of the non-first prediction position with the offset sequence corresponding to the non-first prediction position.

6. The method as described in claim 5, characterized in that, The step of determining the loss based on the first probability distribution and the second probability distribution, and updating the initial draft model based on the loss to obtain a preset draft model includes: The loss corresponding to each of the predicted locations is determined based on the first probability distribution and the second probability distribution of each of the predicted locations; The total loss is obtained by weighting the losses corresponding to each of the predicted positions according to their respective weights. The weight of the loss corresponding to each predicted position is negatively correlated with the depth of the corresponding predicted position. The initial draft model is updated based on the total loss to obtain a preset draft model.

7. The method as described in claim 1, characterized in that, The step of inputting the draft lexical units into the target main model for verification to obtain the text prediction result of the input sequence to be predicted includes: The draft lexical units are grouped in order to obtain several lexical units, and each lexical unit is validated at the same time, with the number of lexical units in each lexical unit increasing sequentially. If there are tuples that fail to be verified in each of the tuples, determine the target tuple that has the smallest number of tuples and fails to be verified, and concatenate the previous tuple of the target tuple with the input sequence to be predicted to obtain a new sequence to be predicted. Return to the step of inputting the input sequence to be predicted and the low-rank adaptation identifier into the preset draft model for prediction to obtain draft words, until there are no failed word groups in each word group, and then concatenate the latest draft word group with the latest input sequence to be predicted as the text prediction result.

8. A draft lexical prediction device, characterized in that, The device includes: The information extraction module is used to extract the input sequence to be predicted and the low-rank adaptation identifier carried in the received prediction request. The lexical acquisition module is used to input the input sequence to be predicted and the low-rank adaptation identifier into a preset draft model for prediction to obtain draft lexical units. The preset draft model includes a gating module and at least two expert modules. The gating module is used to determine the corresponding target expert module in each of the expert modules according to the low-rank adaptation identifier, so that the target expert module can predict the input sequence to be predicted to obtain the draft lexical units. The lexical verification module is used to input the draft lexical into the target main model for verification, and obtain the text prediction result of the input sequence to be predicted. The target main model is obtained by superimposing the adapter corresponding to the low-rank adaptation identifier.

9. A draft lexical prediction device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the draft lexical prediction method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the draft lexical prediction method as described in any one of claims 1 to 7.

Images