Dual-model collaborative reasoning method and electronic equipment

Abstract

The invention discloses a dual-model collaborative reasoning method and electronic equipment, and the method comprises the steps: employing a target model, and carrying out the reasoning based on a first lexical vector; wherein in the reasoning process of the target model, a first hidden layer feature and a second hidden layer feature are output; the semantic information association range represented by the first hidden layer feature is smaller than the semantic information association range represented by the second hidden layer feature; using the target draft model to perform a preset number of times of iterative reasoning once based on the second lexical element vector to output a plurality of candidate lexical element elements; the second lexical vector at least comprises a first hidden layer feature, a second hidden layer feature and at least part of the first lexical vector; wherein the target model performs parallel verification on a plurality of candidate lexical element elements output by the target draft model at one time.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a dual-model collaborative reasoning method and electronic device. Background Technology

[0002] In the field of natural language processing, the EAGLET model is an inference acceleration framework based on a Large Language Model (LLM). EAGLET uses an autoregressive approach to predict multiple tokens using a draft model, which are then input into the target model for verification in parallel. However, the limited prediction accuracy of the draft model results in low-quality predicted tokens, thus increasing the verification burden on the target model. Summary of the Invention

[0003] This application mainly provides a dual-model collaborative reasoning method and electronic device, and the technical solution of this application is implemented as follows.

[0004] In a first aspect, embodiments of this application provide a collaborative reasoning method using two models. The method includes: using a target model to perform reasoning based on a first word vector; wherein, during the reasoning process of the target model, a first hidden layer feature and a second hidden layer feature are output; the first hidden layer feature represents a semantic information association range that is smaller than the semantic information association range represented by the second hidden layer feature; using a target draft model to perform iterative reasoning based on the second word vector for a preset number of times, outputting multiple candidate word elements at once; the second word vector includes at least the first hidden layer feature, the second hidden layer feature, and at least a portion of the first word vector; wherein, the target model performs parallel verification on the multiple candidate word elements output at once by the target draft model.

[0005] Secondly, embodiments of this application provide an electronic device, including: at least one memory configured to store a target model and a target draft model; at least one processor configured to: perform inference based on a first word vector using the target model; wherein, during the inference process of the target model, a first hidden layer feature and a second hidden layer feature are output; the first hidden layer feature represents a semantic information association range smaller than the semantic information association range represented by the second hidden layer feature; using the target draft model, multiple candidate word elements are output at once through a preset number of iterative inferences based on the second word vector; the second word vector includes at least the first hidden layer feature, the second hidden layer feature, and at least a portion of the first word vector; wherein, the target model performs parallel verification on the multiple candidate word elements output at once by the target draft model. Attached Figure Description

[0006] Figure 1 A schematic diagram of a reasoning process provided for an embodiment of this application; Figure 2 A flowchart illustrating a dual-model collaborative reasoning method provided in this application embodiment. Figure 1 ; Figure 3 A flowchart illustrating a dual-model collaborative reasoning method provided in this application embodiment. Figure 2 ; Figure 4 A flowchart illustrating a dual-model collaborative reasoning method provided in this application embodiment. Figure 3 ; Figure 5 A schematic diagram illustrating the training process of an initial draft model provided in this application embodiment; Figure 6 A flowchart illustrating a dual-model collaborative reasoning method provided in this application embodiment. Figure 4 ; Figure 7 A schematic diagram illustrating the output of candidate word meta-elements by a target draft model, provided in an embodiment of this application; Figure 8 A flowchart illustrating a dual-model collaborative reasoning method provided in this application embodiment. Figure 5 ; Figure 9 A flowchart illustrating a dual-model collaborative reasoning method provided in this application embodiment. Figure 6 ; Figure 10 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0007] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0008] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0009] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0010] The following is a description of the relevant technologies used in this application.

[0011] Eaglet is currently a high-performing framework for accelerating large-scale inference on edge devices. This framework uses a small neural network (draft model) to autoregressively predict n future tokens, forming several candidate sequences. These sequences are then fed into the target model in parallel for verification, achieving the effect of "inferring multiple tokens at once." The higher the prediction accuracy of the draft model, the more tokens the target model generates per round of decoding, resulting in better acceleration. Therefore, the key to improving acceleration lies in improving the prediction accuracy of the draft model.

[0012] like Figure 1 As shown, in the traditional inference process of a large model, the inference process is carried out sequentially. The initial input a is provided to the LLM for processing and outputs b. Output b is fed back to the LLM as a new input for processing to generate the next output c. This process continues, using the output of the LLM in each step as the input of the next LLM to generate subsequent results, such as d.

[0013] like Figure 1 As shown, Eaglelet optimizes the inference process by introducing a draft model. The input 'a' is provided to the LLM for processing, generating output 'b'. After obtaining output 'b', Eaglelet uses the draft model to generate multiple predicted tokens, such as... , Further , The inputs are fed into different draft models for parallel processing, generating multiple prediction tokens, including... , This process continues, with multiple parallel draft models based on , Continue output , Thus, the smaller draft model can perform multiple inferences in a shorter period of time, resulting in several predicted tokens, including those mentioned above. , , , , , Then, the sequences are passed to the LLM in parallel for verification, receiving sequences that meet the conditions, such as... , , And, and one or more of these morphemes, for example , This allows for the output of more tokens within the same amount of time.

[0014] However, the existing solutions have two main shortcomings: first, the prediction accuracy of the draft model still has room for improvement; second, the draft model needs to be pre-trained from scratch, requiring a large amount of training data and a considerable amount of training time.

[0015] Based on this, this application provides a dual-model collaborative reasoning method and electronic device. First, during the reasoning process using the target model based on the first word vector, a first hidden layer feature and a second hidden layer feature are output. The first hidden layer feature represents a semantic information association range smaller than that represented by the second hidden layer feature. Thus, during the reasoning process of the target model, hidden layer features representing different semantic information association ranges are output, increasing the diversity of information provided to the draft model. Second, the target draft model performs a preset number of iterative reasonings based on a second word vector containing the first hidden layer feature, the second hidden layer feature, and at least a portion of the first word vector, outputting multiple candidate word elements at once. These are then input into the target model for verification. In this way, using multiple hidden layer features with multi-layered semantic information output by the target model as input to the target draft model increases the effective information input to the target draft model, thereby enhancing the target draft model's ability to understand the context. This allows the target draft model to improve prediction accuracy under limited parameters, significantly improving its prediction performance. Furthermore, combining the multi-round iterative reasoning mechanism with the parallel verification of the target model reduces the overall reasoning time and improves the efficiency of word generation.

[0016] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0017] In one embodiment of this application, as Figure 2 As shown, a dual-model collaborative reasoning method is provided, which may include the following steps S101 to S102.

[0018] S101, using the target model, inference is performed based on the first word vector; during the inference process of the target model, the first hidden layer features and the second hidden layer features are output.

[0019] The semantic information associated with the features in the first hidden layer is smaller than that associated with the features in the second hidden layer.

[0020] In this embodiment, a token is a basic unit of text, such as a word, punctuation mark, or sub-word. The user's input text is divided to obtain the corresponding token set. For example, if the user inputs "The weather is nice today", the corresponding token set may be ["Today", "Weather", "Nice"].

[0021] In this embodiment, a word vector refers to a numerical representation of a word obtained by embedding encoding, used to characterize the semantic information of the word vector. The first word vector is the initial input obtained by segmenting and encoding the user's input text, and may include one or more word vectors encoded based on the corresponding words.

[0022] Hidden layer features refer to intermediate representations obtained by progressively abstracting and extracting input data in a neural network model. Different hidden layer features reflect semantic information at different levels and dimensions. First hidden layer features, also known as shallow hidden layer features, are more focused on basic, surface-level local semantic information, such as lexical structure and grammatical relationships. Second hidden layer features, also known as deep hidden layer features, are more focused on highly abstract, cross-contextual, global semantic information, such as contextual understanding and semantic coherence. In other words, the second hidden layer represents a wider range of semantic channel associations.

[0023] It should be noted that the first hidden layer feature usually corresponds to the hidden layer that is relatively early in the execution order in the target model, while the second hidden layer feature corresponds to the hidden layer that is relatively late in the execution order in the target model.

[0024] In this embodiment, the base model can be any large language model (LLM). The base model may include multiple hidden layers, and during inference based on the first word vector, first hidden layer features and second hidden layer features are output at different hidden layers. Thus, the first and second hidden layer features output by the base model can cover features across multiple semantic dimensions, from local to global.

[0025] S102, using the target draft model, performs a preset number of iterative inferences based on the second word vector to output multiple candidate word elements at once.

[0026] The second word vector includes at least the first hidden layer features, the second hidden layer features, and at least a portion of the first word vector.

[0027] The target model performs parallel verification on multiple candidate lexical elements output by the target draft model at one time.

[0028] In this embodiment, the target draft model is an auxiliary model with a smaller structure and fewer parameters compared to the target model. It is used to perform a preset number of iterative inferences in an autoregressive manner to generate multiple candidate lexical elements for parallel verification by the target model. The iterative inference process of the target draft model may include: the target draft model performs a first round of inference based on the second lexical vector, and then outputs a set of candidate lexical elements. In subsequent inference processes, the multiple lexical elements output by the target draft model in the previous round are input into the target draft model respectively, and the target draft model continues to generate multiple candidate lexical elements until the preset number of iterations is reached. All candidate lexical elements output during the iteration process are output to the target model for parallel verification.

[0029] In the embodiments of this application, during the first round of iteration, the second word vector includes at least the first hidden layer features, the second hidden layer features, and at least a portion of the first word vector. In subsequent iterations, the second word vector may also include relevant features of the candidate word elements output by the target draft model in the previous round.

[0030] In this embodiment, in each round of inference during the iterative process, the target draft model uses both the first hidden layer features and the second hidden layer features as input, enhancing the amount of information input to the target draft model and enabling it to perform better inference, thereby improving the accuracy of the output candidate lexical elements. For example, the first lexical vector includes encodings of lexical elements such as "movie" and "touching." The first hidden layer may capture the basic collocation relationship between "movie" and "touching," while the second hidden layer can identify that the input lexical element is an evaluation of the emotional aspect of the movie. The target draft model uses the combination of the aforementioned first hidden layer features, second hidden layer features, and the first lexical vector as input to the target model as input, which helps the target draft model more accurately predict the next possible lexical element. For example, words containing specific emotional expressions can be multiple candidate lexical elements, such as "deeply" and "heartfelt."

[0031] This application provides a dual-model collaborative reasoning method. First, during the reasoning process using the target model based on the first word vector, a first hidden layer feature and a second hidden layer feature are output. The first hidden layer feature represents a semantic information association range smaller than that represented by the second hidden layer feature. Thus, outputting hidden layer features representing different semantic information association ranges during the target model's reasoning process enhances the diversity of information provided to the draft model. Second, the target draft model performs a predetermined number of iterative reasoning iterations based on a second word vector containing the first hidden layer feature, the second hidden layer feature, and at least a portion of the first word vector, outputting multiple candidate word elements at once. These are then input into the target model for verification. By using multiple hidden layer features with multi-layered semantic information output by the target model as input to the target draft model, the effective information input to the target draft model is increased, thereby enhancing the target draft model's ability to understand the context. This allows the target draft model to improve prediction accuracy with limited parameters, significantly improving its prediction performance. Furthermore, combining the multi-round iterative reasoning mechanism with the parallel verification of the target model reduces the overall reasoning time and improves word generation efficiency.

[0032] In some embodiments, such as Figure 3 As shown, the target model also includes an encoding layer and a language model head layer.

[0033] In this embodiment, the embedding layer is used to encode each input word (which includes multiple words), mapping each word to a corresponding vector form to obtain a word vector. The first word vector is a set of word vectors, including the word vectors corresponding to each word in the input word.

[0034] In this embodiment, the language model head layer (LM head layer) is the last network layer in the target model that transforms hidden states into candidate lexical elements. Here, candidate lexical elements represent one or more lexical elements with relatively high distribution probabilities.

[0035] In some embodiments, before step S101, which uses the target model to perform inference based on the first lexical vector, the method may further include: encoding the input lexical elements based on the coding layer to obtain the first lexical vector.

[0036] In this embodiment of the application, the input lexical element refers to the set of lexical elements obtained after splitting the input text. For example, for the user input text "the weather is good", the corresponding input lexical elements may include one or more of the following: ["weather", "good"].

[0037] For example, such as Figure 3 As shown, for input lexical elements ("weather"), ("Good"), using the encoding layer to encode and map it into a high-dimensional vector, obtain the first word vector, which includes elements corresponding to the input word. Corresponding word vectors and corresponding to the input lexical element Corresponding word vectors .

[0038] In some embodiments, the method may further include: performing probability prediction on the features of the second hidden layer based on the head layer of the language model, and outputting the first candidate word element.

[0039] In this embodiment of the application, for the first word element obtained above (e.g., including...) Figure 3 In , The target model is processed sequentially using multiple hidden layers, with the last hidden layer outputting the features of the second hidden layer. and .

[0040] Furthermore, such as Figure 3 As shown, the first layer of the language model receives features from the second hidden layer. and As input, the tokens undergo linear transformation and normalization to output a probability distribution of a word. Based on this probability distribution, one or more words with the highest probabilities are selected as first candidate word elements. For example... Figure 3 Output a word element As the first candidate lexical element, where for the input lexical element ("weather"), ("Good"), the first candidate meta-element It could be "very".

[0041] This application provides a dual-model collaborative reasoning method. It utilizes an encoding layer to encode input lexical elements, generating a first lexical vector, thereby more effectively extracting semantic information from the input lexical elements and improving the reasoning accuracy of subsequent processes. Furthermore, based on the first lexical vector, it uses multiple hidden layers to output second hidden layer features. These second hidden layer features are used as input to the language model's head layer to output a first candidate lexical element. The second hidden layer features contain a deep semantic understanding of the input lexical elements, which can enhance the language model's head layer's understanding of the input lexical elements and improve the accuracy of the output first candidate lexical element.

[0042] In some embodiments, such as Figure 3 As shown, the target model includes multiple hidden layers.

[0043] In this embodiment, the target model includes N hidden layers, where N is a positive integer greater than or equal to 2. The number of hidden layers is determined based on the actual semantic representation requirements.

[0044] In this process, all hidden layers except the first one take the feature vector output by the previous hidden layer in the inference order as input.

[0045] In some embodiments, such as Figure 4 As shown, for step S101, during the inference process of the target model, outputting the first hidden layer features and the second hidden layer features may include the following steps S201 to S202.

[0046] S201, based on the first hidden layer among multiple hidden layers, reason about the first word vector to obtain the first hidden layer features.

[0047] In this embodiment, the first hidden layer is the hidden layer that is at the beginning of the inference order among multiple hidden layers of the target model. The first hidden layer may include multiple neural network units (e.g., Transformer blocks), and extracts basic semantic information of the first word vector based on the multiple neural network units.

[0048] In this embodiment, the input lexical elements are encoded using an encoding layer to output a first lexical vector, which is then used as the input to the first hidden layer of the target model. The first hidden layer is then used to infer the first lexical vector, outputting the first hidden layer features. This helps the target model capture the local contextual information of the input lexical elements, providing a foundation for semantic processing in subsequent hidden layers.

[0049] S202, based on the last hidden layer among multiple hidden layers, infer the features of the previous hidden layer output by the previous hidden layer to obtain the features of the second hidden layer.

[0050] In this embodiment, the last hidden layer is the last hidden layer in the inference order among multiple hidden layers of the target model. The last hidden layer includes multiple neural network units (e.g., Transformer blocks), and extracts global semantic information from the hidden layer features output by the previous hidden layer based on these multiple neural network units.

[0051] In this embodiment, as described above, the input of the first hidden layer is a first word vector. For each subsequent hidden layer, the feature output by the previous hidden layer is used as the input. This process is repeated sequentially based on the first word vector across multiple hidden layers until the last hidden layer outputs a second hidden layer feature based on the feature output by the previous hidden layer. This second hidden layer feature integrates the knowledge from the reasoning across multiple hidden layers, abstracts global semantic information, and forms a deeper semantic representation.

[0052] This application provides a dual-model collaborative reasoning method. Based on the first hidden layer of a target model, reasoning is performed on the first lexical vector to obtain first hidden layer features, which reflect the local contextual information of the input lexical elements. These first hidden layer features, used as input to the target draft model, enhance the draft model's understanding of local semantic information. Based on the last hidden layer of the target model, reasoning is performed on the previous hidden layer's output features to output second hidden layer features, which reflect the global semantic information of the input lexical elements. These second hidden layer features, used as input to the target draft model, significantly enhance the draft model's understanding of global semantic information. Thus, combining the first and second hidden layer features effectively increases the amount of information acquired by the target draft model, thereby improving its prediction accuracy.

[0053] In some embodiments, such as Figure 3 As shown, the target draft model includes a first draft hidden layer and a second draft hidden layer; wherein, the second draft hidden layer reuses the last hidden layer among multiple hidden layers.

[0054] The target draft model may include multiple draft hidden layers. In order to improve the inference speed, this embodiment of the application will be described using the example of the target draft model including a first draft hidden layer and a second draft hidden layer.

[0055] In this embodiment, the first draft hidden layer is the first hidden layer in the reasoning order of the target draft model. The first draft hidden layer receives a second word vector as input, wherein the second word vector includes at least the features of the first hidden layer, the features of the second hidden layer, and at least a portion of the first word vector. Thus, the first draft hidden layer is used to fuse semantic information at different levels to improve the target draft model's ability to understand the context.

[0056] In this embodiment, the second draft hidden layer is a hidden layer in the target draft model that follows the first draft hidden layer in the reasoning order. The second draft hidden layer receives the feature vector output by the first draft hidden layer as input for reasoning and outputs a new feature vector as input to the language model head layer of the target draft model.

[0057] In this embodiment, the parameters of the second draft hidden layer of the target draft model reuse the parameters of the last hidden layer among the multiple hidden layers of the target model. That is, during training and inference, the parameters of the second draft hidden layer are common to both the target model and the last hidden layer. During inference, inference based on the second draft hidden layer is equivalent to inference based on the last hidden layer among the multiple hidden layers of the target model. The last hidden layer has strong global semantic capture capabilities, enabling it to capture complex semantic features across contexts.

[0058] This application provides a dual-model collaborative reasoning method. The target draft model includes a first draft hidden layer and a second draft hidden layer. The second draft hidden layer reuses the last hidden layer among the multiple hidden layers of the target model. In this way, by sharing parameters, not only is the number of hidden layers in the target draft model increased without changing the total number of model parameters, thereby improving the overall expressive power of the target draft model, but the target draft model can also share the global semantic modeling capability of the last hidden layer of the target model, better collaborate with the target model, and thus improve the prediction accuracy of the target draft model.

[0059] In some embodiments, the target model is obtained by training the initial model based on a first set of training texts; the target draft model is obtained by training the initial draft model based on a second set of training texts.

[0060] In the initial draft model, the second draft hidden layer is obtained by freezing the weight parameters of the last hidden layer in the target model.

[0061] In the embodiments of this application, the first set of training texts and the second set of training texts refer to the corpus data sets used to train the neural network model, which typically include a large number of natural language sentences, paragraphs or documents. The quality and diversity of the training texts affect the performance of the target model and the target draft model obtained after training.

[0062] The first set of training texts is used to train the initial model, and the second set of training texts is used to train the initial draft model. Since the initial draft model is smaller in scale than the initial model, the first set of training texts and the second set of training texts may be different, which helps to improve the training speed of the initial draft model and the initial model.

[0063] During the training phase, an initial model is first constructed. The initial model is then trained based on the first set of training texts to obtain the target model. The structure of the initial model is the same as that of the target model. The training process refers to optimizing the parameters of each layer in the initial model, including the encoding layer, multiple hidden layers, and the language model head layer.

[0064] During training, if the output of the initial model meets the iteration termination condition, such as the difference between the output, the hidden layer features of the intermediate output, and the labeled data is less than a preset threshold, then the initial model is used as the target model.

[0065] Furthermore, an initial draft model is constructed. During the construction process, the weight parameters of the last hidden layer in the target model obtained after training are reused as the weight parameters of the second draft hidden layer of the initial draft model. The weight parameters of the second draft hidden layer are frozen during the training of the initial draft model, using the weight parameters of the last hidden layer without further updates. Instead, the other layers of the initial draft model, including the encoding layer, the first draft hidden layer, and the language model head layer, are updated based on the second training text. By using the weight parameters of the last hidden layer of the target draft model as a priori, the training time of the initial draft model is effectively reduced, and the initial performance of the initial draft model is improved.

[0066] like Figure 5 As shown, in related technologies, during the training of the draft model, the parameters of the draft model are updated based on the loss calculated between the output lexical element b and the labeled data.

[0067] like Figure 5 As shown in this embodiment, during the initial draft model training process, the target model is used as the teacher model for knowledge distillation, based on the candidate word meta-elements output by the draft model. The features output by each hidden layer include , The distribution of lexical elements and hidden layer features corresponding to the output of the target model , The loss is calculated, and the weight parameters of the initial draft model are updated to obtain the target draft model. This allows for more loss calculations during training, resulting in a target draft model with better prediction accuracy after training.

[0068] In some embodiments, such as Figure 3 As shown, the target model and the target draft model can share the weight parameters of the language model head layer. After training the initial model to obtain the target model, the language model head layer in the target model can be used as the language model head layer of the initial draft model, and the weight parameters of the language model head layer are frozen and not updated in subsequent training.

[0069] This application provides a dual-model collaborative reasoning method. It trains an initial model and an initial draft model using different training texts. After training to obtain the target model, the weight parameters of the last hidden layer in the target model are frozen, resulting in a second draft hidden layer in the initial draft model. This allows the initial draft model to directly inherit knowledge from the target model at the start of training, avoiding the instability caused by training the initial draft model from scratch, improving its performance, accelerating its convergence, and reducing its prediction time.

[0070] In some embodiments, for step S102, using the target draft model, multiple candidate word elements are output at once based on the second word vector through a preset number of iterative inferences, such as... Figure 6 As shown, it may include the following steps S301 to S304.

[0071] S301, take the first hidden layer features, the second hidden layer features, at least part of the first word vector and the first coding features corresponding to the first candidate word element as the initial second word vector, use the draft model to perform inference, and output the corresponding third hidden layer features and multiple second candidate word elements.

[0072] In the embodiments of this application, such as Figure 3 As shown, during the first inference of the target draft model, the encoding layer of the target draft model is used to encode the first candidate word element output by the target model based on the first word vector, and the first encoded feature is obtained.

[0073] In this embodiment, the second word vector may include all the first word vectors. In some embodiments, considering the consistency of the input parameter dimensions between the target model and the target draft model, the second word vector includes the first word vector excluding the element furthest from the first candidate word element. All lexical elements other than those in the first lexical vector (in this embodiment, taking two input lexical elements of the target model as an example, the second lexical vector may include those from the first lexical vector). ), and the first encoded feature corresponding to the first candidate word element output by the target model inference.

[0074] In the embodiments of this application, such as Figure 3 As shown, in the first inference of the target draft model, the initial second word vector includes the first hidden layer features output by the target model during the inference process based on the first word vector (including...). and ), second hidden layer features (including and ), at least some of the first word vectors (including ) and the first candidate meta-element The corresponding first coding feature The target draft model fuses the initial second word vector based on a fully connected layer (FCLayer), and uses the feature vector output by the FCLayer as the input to the first draft hidden layer. Inference is performed using the first and second draft hidden layers, and during the inference process, the features of the third hidden layer (including...) are output. Figure 3 In , ) and multiple second candidate meta-elements (including Figure 3 In and ).

[0075] For example, refer to Figure 3 Input word elements For "today", enter the word element. For the input word element, the target model encodes it based on the first word vector corresponding to the input word element and outputs the first candidate word element. For "weather"; in the first inference of the target draft model, based on the above initial second word vector, the output second candidate word element can include ("forecast"), ("very").

[0076] S302, update the initial second word vector by using the features of the third hidden layer and the second encoding features corresponding to each of the multiple second candidate word elements, and determine multiple updated second word vectors.

[0077] In the embodiments of this application, after the first inference of the target draft model is completed, the first inference of the target draft model is taken as the previous inference, and the second inference of the target draft model is taken as the current inference.

[0078] In the current inference, the third hidden layer feature output by the target draft model in the previous inference process is added to the initial second word vector; and in the current inference, the encoding layer of the target draft model is used to encode any one of the second candidate word elements output by the target draft model at the end of the previous inference process to obtain the corresponding second encoding feature, and the second encoding feature is added to the initial second word vector to obtain an updated second word vector.

[0079] For the initial word vector, update the initial word vector based on the features of the third hidden layer and the second encoding features corresponding to a second candidate word element to obtain an updated second word vector. Following the same method, for each second candidate word element, a corresponding updated second word vector is obtained. Thus, for multiple second candidate word elements, multiple updated second word vectors are obtained for each element.

[0080] S303 utilizes the target draft model to perform parallel inference based on multiple updated second word vectors, outputting multiple third hidden layer features and multiple third candidate word elements.

[0081] In this embodiment, multiple updated second word vectors are input into a target draft model, which then performs parallel inference on the different updated second word vectors. Specifically, for each inference operation based on the updated second word vectors, the target draft model outputs third hidden layer features and at least two third candidate word elements during the inference process. Thus, for multiple updated second word vectors, the target draft model outputs multiple third hidden layer features and multiple third candidate word elements after parallel inference.

[0082] For example, such as Figure 3 As shown, the second candidate word meta-element can include and For the second candidate meta-element The encoder of the target draft model encodes it to obtain the corresponding second encoded features. And based on the second coding feature During the inference process, the third hidden layer features are output (including...) , ), and the second encoded feature and third hidden layer features (including) , ) are added to the aforementioned initial second word vector to obtain the corresponding updated second word vector; for the second candidate word element Referring to the aforementioned second candidate word meta-elements The corresponding second coding feature is obtained in this way. The features of the third hidden layer are added to the aforementioned initial second word vector to obtain the corresponding updated second word vector.

[0083] like Figure 7 As shown, the second candidate word meta-elements include and For each second candidate lexical element, the corresponding updated second lexical vector is obtained and input into the target draft model for inference. (“forecast”), the target draft model outputs multiple corresponding third candidate word meta-elements, for example, may include ("Broadcast") and (“say”); for the second candidate meta-element ("very"), the target draft model outputs multiple third candidate word meta-elements, for example, may include ("Sunny") and ("good").

[0084] S304, taking the third candidate lexical element as the second candidate lexical element, iteratively determining multiple steps of updating the second lexical vector and using the target draft model for parallel inference, until the preset number of iterations is reached, and multiple candidate lexical elements are obtained.

[0085] In this embodiment of the application, when performing subsequent inference based on the target draft model, for the current inference, the multiple third candidate word elements output by the target draft model in the previous inference are used as second candidate word elements. The process of iterating from step S302 to step S303 includes updating the initial second word vector based on the multiple third hidden layer features and multiple second candidate word elements obtained from the previous inference, to obtain multiple updated second word vectors corresponding to the multiple second candidate word elements respectively; and performing parallel inference on the multiple updated second word vectors based on the target draft model to complete the current inference, outputting new third hidden layer features and multiple third candidate word elements as input for the next inference.

[0086] It should be noted that the above iteration process will continue to run until the preset number of iterations is reached. The preset number of iterations can be set according to the specific application scenario to achieve a balance between the accuracy and performance of the model output, for example, it can be 3-5 iterations.

[0087] In this embodiment, after the iteration reaches a preset number of times, the set of the second or third candidate word elements output by the target draft model in each inference is taken as multiple candidate word elements, and the multiple candidate word elements are further input into the target model for parallel verification.

[0088] This application provides a dual-model collaborative reasoning method. In the first reasoning of the target draft model, an initial second word vector is used as input for reasoning, outputting the corresponding third hidden layer features and multiple second candidate word elements. In subsequent reasoning, the third hidden layer features and third candidate word elements output from the previous reasoning of the target draft model are added to the initial second word vector to obtain an updated second word vector. Parallel reasoning is then performed based on multiple updated word vectors, iterating the parallel reasoning steps of the target draft model until a preset number of iterations. In this way, the hidden layer features and candidate word elements from the previous reasoning of the target draft model can be added to the next reasoning. Richer input information helps the target draft model output more accurate and context-sensitive results, thereby improving the accuracy of the candidate word elements output by the target draft model.

[0089] In some embodiments, the target draft model further includes a fully connected layer and a language model header.

[0090] In this embodiment, the fully connected layer (FC Layer) is used to receive the initial second word vector (in the first inference of the target draft model) or update the second word vector (in the second to a preset number of inferences of the target draft model), and jointly encode each feature in the input second word vector to generate a fused feature vector.

[0091] In this embodiment of the application, the language model head layer (LM head layer) of the target draft model is used to receive the feature vector output from the second draft hidden layer, convert the received feature vector into a probability distribution of possible tokens, and output one or more of the tokens with the highest probability as candidate lexical elements.

[0092] In the embodiments of this application, such as Figure 3 As shown, the target draft model may also include an embedding layer, which is a neural network structure in the target draft model used to encode the input candidate lexical elements and convert them into a semantic representation of a high-dimensional vector.

[0093] It should be noted that the weight parameters of the encoding layer of the target draft model can be reused from the weight parameters of the encoding layer of the target model; the weight parameters of the language model head layer of the target draft model can be reused from the weight parameters of the language model head layer of the target model.

[0094] In some embodiments, such as Figure 8 As shown, for step S303, using the target draft model, parallel inference is performed based on multiple updated second word vectors to output multiple third hidden layer features and multiple third candidate word elements, which may include the following steps S401 to S403.

[0095] S401, based on the target draft model, uses a fully connected layer to process multiple updated second word vectors in parallel to obtain multiple corresponding fused feature vectors.

[0096] In this embodiment of the application, for the first inference of the target draft model, the initial second word vector (including the first hidden layer features) is used. and Second hidden layer features and At least some of the first word vectors and the first candidate meta-element The corresponding first coding feature The different input features are integrated into a unified feature representation to obtain the corresponding fused feature vector, thereby enhancing the target draft model's ability to understand the context.

[0097] In this embodiment, for subsequent inference of the target draft model, the target draft model performs parallel inference based on multiple updated second word vectors. For each updated second word vector, the fully connected layer of the target draft model fuses and jointly encodes different features in the updated second word vector, outputting a corresponding fused feature vector. Thus, for multiple updated second word vectors, the target draft model performs parallel fusion processing to obtain multiple fused feature vectors corresponding to each of the multiple updated second word vectors.

[0098] S402, based on the first draft hidden layer, the second draft hidden layer and multiple fused feature vectors, obtain multiple corresponding third hidden layer features.

[0099] In the embodiments of this application, such as Figure 3 As shown, based on the first draft hidden layer and the second draft hidden layer, inference is performed on the multiple fused feature vectors output by the fully connected layer, and the output is the multiple third hidden layer features corresponding to the multiple updated second word vectors.

[0100] S403, based on the parallel inference of multiple third hidden layer features in the head layer of the language model, obtains multiple third candidate word elements.

[0101] In this embodiment, the language model head layer is a neural network structure at the end of the inference process of the target draft model. It is used to perform parallel inference on the third hidden layer features output by the second draft hidden layer and output multiple third candidate word elements corresponding to multiple updated second word vectors.

[0102] For example, in a text completion task, the language model head layer can predict the next most likely word to appear based on the semantic features of the current context, and generate multiple third candidate word elements for the verification of the target model.

[0103] This application provides a dual-model collaborative reasoning method. It fuses the input updated second word vector through a fully connected layer and combines the third hidden layer features output from the first and second draft hidden layers. The language model head layer then performs parallel reasoning on multiple third hidden layer features to obtain multiple third candidate word elements. Thus, during the reasoning process of the target draft model, the third hidden layer features output from the hidden layers are obtained. These features can serve as input features for the next reasoning step, expanding the dimensionality of semantic information acquired by the target draft model in the next reasoning step. Furthermore, by processing the fused feature vector based on the first and second draft hidden layers, the third hidden layer features contain more semantic feature information, thereby improving the prediction accuracy of the target draft model.

[0104] In some embodiments, the third hidden layer features include shallow hidden layer features and deep hidden layer features.

[0105] In this embodiment, the shallow hidden layer features are basic, surface-level local semantic features based on the output of the first draft hidden layer, including more specific language structure information, such as syntactic components and part-of-speech tagging.

[0106] In this embodiment, the deep hidden layer features are highly abstract, cross-context global semantic features based on the output of the second draft hidden layer, such as the topic and intent of the input lexical elements.

[0107] For example, shallow hidden layer features may include the topic or sentiment of the input second lexical vector, while deep hidden layer features may be more complex syntactic structures or semantic relationships obtained by further reasoning from the second draft hidden layer based on the shallow hidden layer features.

[0108] In some embodiments, such as Figure 9 As shown, for step S402, based on the first draft hidden layer, the second draft hidden layer, and multiple fused feature vectors, multiple corresponding third hidden layer features are obtained, which may include the following steps S501 to S502.

[0109] S501, based on the first draft hidden layer, reason about the fused feature vector to obtain the shallow hidden layer features.

[0110] In this embodiment, the first draft hidden layer is the first hidden layer in the target draft model, which is used to further infer the fused feature vector output by the fully connected layer and output shallow hidden layer features.

[0111] The first draft hidden layer is used to perform basic local semantic information reasoning on the fused feature vector, such as lexical relationships and grammatical structures. Its output shallow hidden layer features are a shallow semantic representation with more context awareness, which can serve as the basis for deep reasoning in the subsequent second draft hidden layer.

[0112] S502, based on the features of the shallow hidden layer inference based on the second draft hidden layer, the features of the deep hidden layer are obtained.

[0113] In this embodiment, the second draft hidden layer is a hidden layer that follows the first draft hidden layer in the reasoning order. It is used to further reason about the shallow hidden layer features to extract higher-level abstract semantic information and output deep hidden layer features. Compared to shallow hidden layer features, deep hidden layer features contain more abstract global semantic information and are closer to the output style of the target model. This characteristic helps to improve the matching degree between the draft model and the target model, and this improved matching degree can further increase the acceptance rate in the validation stage.

[0114] As mentioned earlier, the weight parameters of the second draft hidden layer are obtained by reusing the weight parameters of the last hidden layer of the target model. In this way, without increasing the total number of parameters of the model, a two-layer hidden layer is constructed in the target draft model. The input second word vector is inferred multiple times, and the corresponding hidden layer features are output after each inference of the draft hidden layer as input for the next inference of the target draft model. In this way, richer features can be output to the next inference of the target draft model, thereby improving the prediction accuracy of the target draft model.

[0115] This application provides a dual-model collaborative inference method. A first draft hidden layer infers the fused feature vector, outputting shallow hidden layer features that represent the local semantic expression of the input second word vector at a shallower level. A second draft hidden layer infers the shallow hidden layer features, outputting deep hidden layer features that represent the global semantic expression of the input second word vector at a deeper level. Thus, through this dual-hidden-layer structure, the inference process of the target draft model improves the model's understanding and performance of the input second word vector through multiple inferences based on the hidden layers without increasing the total number of parameters, thereby enhancing the accuracy and stability of the target draft model's predictions.

[0116] In yet another embodiment of this application, as Figure 10 As shown, an electronic device 100 is provided, comprising: At least one memory 1001 is configured to store the target model and the target draft model; At least one processor 1002 is configured as follows: Using the target model, inference is performed based on the first word vector; during the inference process of the target model, the first hidden layer features and the second hidden layer features are output; the semantic information association range represented by the first hidden layer features is smaller than the semantic information association range represented by the second hidden layer features; Using the target draft model, multiple candidate word elements are output at once through a predetermined number of iterative inferences based on the second word vector; the second word vector includes at least the first hidden layer features, the second hidden layer features, and at least a portion of the first word vector; wherein, the target model performs parallel verification on the multiple candidate word elements output at once by the target draft model.

[0117] In some embodiments, such as Figure 3 As shown, the target model includes multiple hidden layers; the draft model includes a first draft hidden layer and a second draft hidden layer; wherein, the second draft hidden layer reuses the last hidden layer among the multiple hidden layers.

[0118] In some embodiments, at least one processor 1002 is further configured to: encode input lexical elements based on the coding layer to obtain a first lexical vector.

[0119] In some embodiments, at least one processor 1002 is further configured to: perform probabilistic prediction on the features of the second hidden layer based on the head layer of the language model, and output a first candidate lexical element.

[0120] In some embodiments, at least one processor 1002 is further configured to: infer a first word vector based on a first hidden layer among a plurality of hidden layers to obtain a first hidden layer feature; and infer a previous hidden layer feature output by a previous hidden layer based on the last hidden layer among a plurality of hidden layers to obtain a second hidden layer feature.

[0121] In some embodiments, the target model is obtained by training the initial model based on the first set of training texts; the target draft model is obtained by training the initial draft model based on the second set of training texts; wherein, the second draft hidden layer in the initial draft model is obtained by freezing the weight parameters of the last hidden layer in the target model.

[0122] In some embodiments, at least one processor 1002 is further configured to: use the first hidden layer features, the second hidden layer features, at least a portion of the first lexical vectors, and the first encoding features corresponding to the first candidate lexical elements as initial second lexical vectors; perform inference using a draft model to output corresponding third hidden layer features and multiple second candidate lexical elements; update the initial second lexical vectors with the second encoding features corresponding to the third hidden layer features and the multiple second candidate lexical elements respectively to determine multiple updated second lexical vectors; perform parallel inference using a target draft model based on the multiple updated second lexical vectors respectively to output corresponding multiple third hidden layer features and multiple third candidate lexical elements; use the third candidate lexical elements as second candidate lexical elements, iteratively determine multiple updated second lexical vectors and perform parallel inference using the target draft model, until the iteration reaches a preset number, and obtain multiple candidate lexical elements.

[0123] In some embodiments, the target draft model further includes an encoding layer and a language model header.

[0124] In some embodiments, at least one processor 1002 is further configured to: perform parallel fusion processing of multiple updated second word vectors based on the coding layer of the target draft model to obtain multiple corresponding fused feature vectors; obtain multiple corresponding third hidden layer features based on the first draft hidden layer, the second draft hidden layer, and the multiple fused feature vectors; and perform parallel inference on the multiple third hidden layer features based on the language model head layer to obtain multiple third candidate word elements.

[0125] In some embodiments, the third hidden layer features include shallow hidden layer features and deep hidden layer features.

[0126] In some embodiments, at least one processor 1002 is further configured to: infer the fused feature vector based on a first draft hidden layer to obtain shallow hidden layer features; and infer the shallow hidden layer features based on a second draft hidden layer to obtain deep hidden layer features.

[0127] The processing method provided in the embodiments of this application will be described in detail below with reference to specific application scenarios.

[0128] Eaglet is currently the latest and most effective framework for accelerating inference on large edge models. This framework uses a relatively small neural network (draft model) to autoregressively guess multiple future tokens, forming several candidate sequences. These sequences are then fed into the target model in parallel for verification, achieving the effect of "inferring multiple tokens at once." The higher the accuracy of the draft model's guesses, the more tokens the target model generates in each decoding round, resulting in better acceleration. Therefore, the key to improving acceleration lies in improving the prediction accuracy of the draft model.

[0129] like Figure 1 As shown, the Eagleet inference acceleration framework differs from traditional large model inference methods in that it uses a very small draft model for the target model (LLM), and performs multiple inferences in a short period of time to obtain several candidate sequences. These sequences are then passed into the target model in parallel for verification, and the sequences that meet the conditions are received, thereby achieving the effect of outputting more tokens in the same amount of time.

[0130] The Eaglet inference acceleration framework mentioned above has two main shortcomings: first, the prediction accuracy of the draft model still has room for improvement; second, the draft model needs to be pre-trained from scratch, requiring a large amount of training data and a considerable amount of training time.

[0131] Based on this, in this embodiment of the application, the input features and inference process of the draft model in the Eaglet inference acceleration framework are improved. The hidden state features (second word vector) input to the target draft model include the first hidden layer features and the second hidden layer features output by the target model. Meanwhile, in terms of inference, after the target draft model outputs shallow hidden layer features through the first draft hidden layer, it inputs them to the second draft hidden layer to output deep hidden layer features, and then enters the LM head layer of the target draft model.

[0132] In the embodiments of this application, such as Figure 3 As shown, the input to the target draft model includes embedding vectors. (Including at least a portion of the first word vector and the first candidate word element output by the target model, or at least a portion of the first word vector and the second candidate word element output by the previous target draft model inference), and the first hidden layer feature output by the first hidden layer among multiple hidden layers. The second hidden layer feature output by the last hidden layer in a multi-hidden-layer sequence. The reason is: the shallow hidden states of the language model (features of the first hidden layer). ) and deep hidden state (second hidden layer features) The hidden states contain semantic information of different dimensions. Shallow hidden states tend to contain basic, surface-level, local semantic information, while deep hidden states tend to contain more abstract, cross-context, global semantic information. Fusing these two types of hidden states with the embedding vector can increase the amount of effective information in the input, thereby improving the prediction accuracy of the target draft model.

[0133] During the inference phase, in each inference round, when the target draft model infers for the first time, it outputs the first hidden layer features of the target model. (include and ) and second hidden layer features (include and Output shallow hidden layer features Then input it into the second draft hidden layer (reusing the last hidden layer of the target model), and output the deep hidden layer features. Then input it into the LM head to obtain the final prediction result, the second candidate word element.

[0134] Furthermore, starting from the second inference, the target draft model is input with the shallow hidden layer features output from the previous inference. and deep hidden layer features This method is equivalent to improving the overall expressive power of the target draft model (effectively having two hidden layers) while keeping the number of parameters constant, thereby improving prediction accuracy.

[0135] In addition, such as Figure 5 As shown, when training the initial draft model, the features of the shallow hidden layers are calculated. Deep hidden layer features The loss of candidate word elements, along with the last hidden layer in the target model, can be regarded as the second draft hidden layer of the target draft model. The frozen parameters, as prior knowledge, can play a "cold start" role, providing a basic direction for updating the initial draft model. Therefore, compared to training the initial draft model from scratch, it can effectively reduce the amount of data and training time required for training.

[0136] Thus, in this embodiment of the application, the average prediction accuracy of the target draft model is effectively improved while the amount of data and training time required for training the target draft model are reduced, all without changing the total number of parameters.

[0137] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

[0138] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0139] It should be understood that the descriptions of the above storage medium and device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0140] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0141] It should also be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0142] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0143] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0144] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0145] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0146] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A dual-model collaborative reasoning method, the method comprising: Using the target model, inference is performed based on the first word vector; wherein, during the inference process of the target model, the first hidden layer features and the second hidden layer features are output; the semantic information associated range represented by the first hidden layer features is smaller than the semantic information associated range represented by the second hidden layer features; Using a target draft model, multiple candidate word elements are output at once through a preset number of iterative inferences based on a second word vector; the second word vector includes at least the first hidden layer features, the second hidden layer features, and at least a portion of the first word vector; wherein, the target model performs parallel verification on the multiple candidate word elements output at once by the target draft model.

2. The method according to claim 1, wherein the target model further comprises an encoding layer and a language model head layer; Before performing inference based on the first word vector using the target model, the method further includes: The first word vector is obtained by encoding the input word elements based on the encoding layer. The method further includes: Based on the language model head layer, the features of the second hidden layer are probabilistically predicted, and the first candidate word element is output.

3. The method according to claim 1, wherein the target model comprises multiple hidden layers; During the inference process of the target model, the output of the first hidden layer features and the second hidden layer features includes: Based on the first hidden layer among the plurality of hidden layers, the first word vector is inferred to obtain the first hidden layer features; Based on the last hidden layer among the multiple hidden layers, the features of the previous hidden layer are inferred from the output of the previous hidden layer to obtain the features of the second hidden layer.

4. The method according to claim 3, The target draft model includes a first draft hidden layer and a second draft hidden layer; wherein... The second draft hidden layer reuses the last hidden layer among the plurality of hidden layers.

5. The method according to claim 4, The target model is obtained by training the initial model based on the first set of training texts; The target draft model is obtained by training the initial draft model based on the second set of training texts; in, The second draft hidden layer in the initial draft model is obtained by freezing the weight parameters of the last hidden layer in the target model.

6. The method according to claim 1, wherein the step of using the target draft model to perform a preset number of iterative inferences based on the second lexical vector to output multiple candidate lexical elements at once includes: The first hidden layer features, the second hidden layer features, at least a portion of the first word vectors and the first candidate word elements are used as the first encoded features corresponding to the first word vectors. The draft model is used to perform inference to output the corresponding third hidden layer features and multiple second candidate word elements. The initial second word vector is updated by using the third hidden layer features and the second encoding features corresponding to each of the multiple second candidate word elements, thereby determining multiple updated second word vectors; Using the target draft model, parallel inference is performed based on the multiple updated second word vectors to output the corresponding multiple third hidden layer features and multiple third candidate word elements; Using the third candidate lexical element as the second candidate lexical element, iterate through the steps of determining multiple updated second lexical vectors and performing parallel inference using the target draft model until the iteration reaches a preset number, thereby obtaining multiple candidate lexical elements.

7. The method according to claim 6, wherein the target draft model further comprises a fully connected layer and a language model head layer; The process utilizes the target draft model to perform parallel inference based on the multiple updated second word vectors, outputting corresponding third hidden layer features and multiple third candidate word elements, including: Based on the target draft model, the fully connected layer performs parallel fusion processing on the multiple updated second word vectors to obtain the corresponding multiple fused feature vectors. Based on the first draft hidden layer, the second draft hidden layer, and the multiple fused feature vectors, multiple corresponding features of the third hidden layer are obtained. Multiple third candidate lexical elements are obtained by parallel inference of multiple third hidden layer features based on the language model head layer.

8. The method according to claim 7, wherein the third hidden layer features include shallow hidden layer features and deep hidden layer features; The process of obtaining multiple corresponding third hidden layer features based on the first draft hidden layer, the second draft hidden layer, and the multiple fused feature vectors includes: Based on the first draft hidden layer, the fused feature vector is inferred to obtain the shallow hidden layer features; Based on the second draft hidden layer, the features of the shallow hidden layer are inferred to obtain the features of the deep hidden layer.

9. An electronic device, comprising: At least one memory is configured to store the target model and the target draft model; At least one processor is configured as follows: Using the target model, inference is performed based on the first word vector; wherein, during the inference process of the target model, the first hidden layer features and the second hidden layer features are output; the semantic information associated range represented by the first hidden layer features is smaller than the semantic information associated range represented by the second hidden layer features; Using a target draft model, multiple candidate word elements are output at once through a preset number of iterative inferences based on a second word vector; the second word vector includes at least the first hidden layer features, the second hidden layer features, and at least a portion of the first word vector; wherein, the target model performs parallel verification on the multiple candidate word elements output at once by the target draft model.

10. The electronic device according to claim 9, The target model includes multiple hidden layers; The draft model includes a first draft hidden layer and a second draft hidden layer; in, The second draft hidden layer reuses the last hidden layer among the plurality of hidden layers.

Images