Self-speculation decoding method and device based on dual advance quit, medium and equipment

Abstract

The invention relates to the technical field of natural language processing, can be applied to business scenes such as financial analysis and medical report generation, and provides a self-speculation decoding method and device based on dual advanced exit, a medium and equipment. Connecting a lightweight adapter and reusing a language modeling head of the lightweight adapter to construct a self-draft model; for a to-be-decoded sequence, when the main model is calculated to the lth layer, executing hierarchy advance quit, obtaining a hidden state and inputting the hidden state into the self-drafting model; a candidate sequence is generated through iteration of the self-drafting model, and if the highest probability of semantic unit generation in a certain step is lower than a threshold value, semantic unit-level pre-exit iteration termination is executed; splicing the middle hidden state corresponding to each iteration step into a parallel tensor, and inputting the parallel tensor into a remaining layer of the main model for one-time parallel verification to obtain a verification result of each position; and determining a final decoding sequence according to the verification result and the candidate sequence. According to the embodiment, the deployment and calculation overhead is reduced, and the end-to-end reasoning efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of natural language processing, in particular, to a self-speculation decoding method and device based on double early exit, a medium and equipment. BACKGROUND

[0002] In recent years, in the field of finance, intelligent investment advice, automatic report generation, and in the field of medicine, auxiliary diagnosis text generation, medical record structured summary, etc., using large language models for content creation and decision support has become an industry trend. However, as the scale of large language models (LLMs) continues to expand, its effectiveness in text generation tasks has improved significantly, but it has also brought huge computational burden and inference delay. Especially in the self-recurrence generation scene, the model needs to generate each token sequentially, resulting in limited overall throughput, making it difficult to meet the actual application requirements of high concurrency and low delay.

[0003] In related technologies, mainly relying on an independently trained draft model, there are inherent disadvantages of high deployment cost and difficulty in aligning with the main model, which is a huge cost and operation pressure for financial or medical institutions that need to deploy multiple specialized models. Subsequent bootstrap solutions such as Medusa or retrieval-based methods, while avoiding separate training, often struggle to balance draft delay and acceptance rate: additional parameters or retrieval overhead weaken the acceleration benefits; and the self-drafting method based on early exit cannot adapt to the dynamic difficulty of samples if the design is not properly designed, wasting computing power on simple text and not drafting enough in complex reasoning, resulting in suboptimal end-to-end efficiency. SUMMARY

[0004] The embodiments of the present disclosure at least provide a self-speculation decoding method, device, medium and equipment based on double early exit, which significantly reduces deployment and computing overhead while ensuring consistent generation quality and baseline model distribution, and improves end-to-end inference efficiency.

[0005] The embodiments of the present disclosure provide a self-speculation decoding method based on double early exit, comprising: cutting the first l layers of a preset decoding main model as a shallow subnetwork, connecting a lightweight adapter to the output end of the shallow subnetwork, and connecting the language modeling head of the preset decoding main model to the output end of the lightweight adapter to build a self-drafting model; obtaining a to-be-decoded sequence and inputting the to-be-decoded sequence into the preset decoding main model, performing a first heavy level early exit when calculating to the first l layer, obtaining the hidden state output by the first l layer; and inputting the hidden state into the self-drafting model; based on the input hidden state, performing multi-step iteration by the self-drafting model to generate a candidate semantic unit sequence; during the multi-step iteration, if the highest probability of the next semantic unit predicted by the current iteration step is lower than a preset confidence threshold, performing second heavy semantic unit level early exit to stop iteration; concatenating the intermediate hidden states output by the shallow sub-network corresponding to each iteration step in the multi-step iteration process of the self-drafting model to form a parallel tensor; inputting the parallel tensor to the remaining network layers after the lth layer of the preset decoding main model to perform parallel forward calculation to obtain a verification result for each position in the candidate semantic unit sequence; determining a target decoding sequence according to the verification result and the order comparison result of the candidate semantic unit sequence.

[0006] The embodiment of the disclosure provides a self-speculation decoding device based on double early exit, comprising: a model construction module configured to intercept the first l layers of a preset decoding main model as a shallow sub-network, connect a light adapter to an output end of the shallow sub-network, and connect a language modeling head of the preset decoding main model to an output end of the light adapter to construct a self-drafting model; a drafting triggering module configured to obtain a to-be-decoded sequence, input the to-be-decoded sequence into the preset decoding main model, perform first heavy layer level early exit when calculating to the lth layer to obtain a hidden state output by the lth layer, and input the hidden state into the self-drafting model; a semantic drafting module configured to perform multi-step iteration by the self-drafting model based on the input hidden state to generate a candidate semantic unit sequence; during the multi-step iteration, if the highest probability of the next semantic unit predicted by the current iteration step is lower than a preset confidence threshold, performing second heavy semantic unit level early exit to stop iteration; a parallel verification module configured to concatenate the intermediate hidden states output by the shallow sub-network corresponding to each iteration step in the multi-step iteration process of the self-drafting model to form a parallel tensor; inputting the parallel tensor to the remaining network layers after the lth layer of the preset decoding main model to perform parallel forward calculation to obtain a verification result for each position in the candidate semantic unit sequence; a sequence determination module configured to determine a target decoding sequence according to the verification result and the order comparison result of the candidate semantic unit sequence.

[0007] The embodiment of the present disclosure provides a computer device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the self-speculation decoding method based on double early exit as described in any possible embodiment.

[0008] The embodiment of the present disclosure provides a computer readable storage medium, which stores a computer program, and the computer program is run by a processor to implement the self-speculation decoding method based on double early exit as described in any possible embodiment.

[0009] The self-speculation decoding method based on double early exit, the device, the medium and the equipment provided in the embodiment of the present disclosure, specifically, through the first heavy level early exit mechanism, the hidden state is obtained and input into the self-drafting model when the preset decoding main model is calculated to a specific layer, the unnecessary calculation amount of the main model is reduced, the overall decoding efficiency is effectively improved, through the second heavy semantic unit level early exit mechanism, the iteration is dynamically stopped according to the probability of the predicted semantic unit in the iteration process of the self-drafting model, the invalid iteration step is avoided, and the calculation resource consumption is further reduced, the intermediate hidden state generated by the multi-step iteration of the self-drafting model is spliced into a parallel tensor and input into the remaining network layer of the main model for parallel calculation, the calculation resources can be fully utilized, the verification process is accelerated, and the accuracy of the verification result is guaranteed, and finally, the target decoding sequence is determined according to the comparison between the verification result and the candidate semantic unit sequence, the calculation efficiency and the resource utilization rate are significantly improved on the premise of ensuring the decoding quality.

[0010] In this way, the present disclosure realizes the self-speculation decoding without the independent draft model and the dynamic control of the drafting length by constructing the self-drafting model composed of the shallow sub-network of the main model and the light adapter and based on the synergistic effect of the double early exit mechanism and the parallel calculation strategy, the deployment and the calculation overhead are significantly reduced on the premise of ensuring the generation quality and the consistency with the distribution of the baseline model, and the end-to-end inference efficiency is improved.

[0011] In order to make the above-mentioned purpose, features and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are specifically described, and the accompanying drawings are shown as follows. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required to be cited in the embodiments will be briefly introduced hereinafter, which are incorporated into the description and form a part of the description, and show the embodiments consistent with the present disclosure, and are used to explain the technical solutions of the present disclosure together with the description. It should be understood that the following drawings only show some of the embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0013] Figure 1 A schematic diagram of an application environment of a self-speculation decoding method based on double early exit provided by an embodiment of the present disclosure is shown. Figure 2 A flowchart of a self-speculation decoding method based on double early exit provided by an embodiment of the present disclosure is shown. Figure 3 A flowchart of a self-drafting model and preset decoding master model alignment method provided by an embodiment of the present disclosure is shown. Figure 4 A flowchart of a self-drafting model multi-step iteration method provided by an embodiment of the present disclosure is shown. Figure 5 A flowchart of a preset decoding master model parallel verification method provided by an embodiment of the present disclosure is shown. Figure 6 A flowchart of a sequence alignment method provided by an embodiment of the present disclosure is shown. Figure 7 A structural schematic diagram of a self-speculation decoding device based on double early exit provided by an embodiment of the present disclosure is shown. Figure 8 A structural schematic diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0014] In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. The components of the embodiments of the present disclosure described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present disclosure.

[0015] It should be noted that like reference numerals and characters refer to like items throughout the attached drawings and alternative embodiments thereof, noting that, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0016] The term "and / or", used herein, merely describes an associated relationship, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0017] To facilitate the understanding of the present embodiment, first, the execution subject of the self-speculation decoding method based on double early exit provided by the present embodiment is described in detail. The self-speculation decoding method based on double early exit provided by the present embodiment can be applied in the application environment as shown in Figure 1 , wherein the client communicates with the server through the network. The client can be a mobile device, a user terminal, a terminal, a handheld device, a computing device, etc. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud storage, big data, and artificial intelligence platform.

[0018] The self-speculation decoding method based on double early exit provided by the present embodiment will be described in detail below in combination with the drawings. Referring to Figure 2 , a flowchart of a self-speculation decoding method based on double early exit provided by the present embodiment is shown, which comprises the following S201-S205: S201, intercepting the first l-layer network of the preset decoding main model as a shallow subnetwork, connecting a lightweight adapter to the output end of the shallow subnetwork, and connecting the language modeling head of the preset decoding main model to the output end of the lightweight adapter to construct a self-drafting model.

[0019] It can be understood that the preset decoding main model refers to a large language model that has been trained for autoregressive text generation, which can usually be constructed based on the Transformer architecture, containing tens of billions of parameters and tens of sequentially connected decoding layers, for example, a model with 7 billion or 13 billion parameters; the self-drafting model refers to a lightweight sub-model constructed based on the main model components for quickly generating multiple candidate subsequent semantic units in the present disclosure.

[0020] Here, in the financial service or analysis scenario, the preset decoding main model can be a large language model trained on financial corpus, used for processing text generation tasks such as risk assessment report generation, prospectus summary, or market sentiment analysis; or in the medical health and clinical assistance scenario, the preset decoding main model can be a large language model trained on medical literature and medical record data, used to assist in generating diagnostic report summaries, patient education materials, or preliminary drafts of scientific research literature. It should be noted that the above fields and uses are only examples, and the specific architecture, parameter size, training data, and application field of the preset decoding main model are not limited in this regard. In actual implementation, they can be selected and adapted according to task requirements, computing resources, and performance indicators.

[0021] Specifically, when constructing the self-drafting model, the first l layers of the network can be intercepted from the preset decoding main model to form a shallow sub-network, which is responsible for preliminary feature extraction and representation of the input text. A lightweight adapter is connected to the output end of the shallow sub-network, and then the original language modeling head of the preset decoding main model, which is responsible for mapping high-level features to the final layer of the vocabulary probability distribution, is connected to the output end of the lightweight adapter. The value of l in the l layers of the network can be set comprehensively according to the total number of layers of the model, the requirements for drafting speed and quality, and the constraints of computing resources. For example, it can be set to an integer between 1 / 16 and 1 / 10 of the total number of layers, or in a model with a hundred billion parameters, it is often set to 2 or 3. The lightweight adapter is used to learn and bridge the distribution difference between the shallow features and deep semantic representations of the main model during the training phase, so that the output probability distribution of the self-drafting model approximates the final output distribution of the main model. For example, in a preset decoding main model for text generation, assuming that the preset decoding main model has 10 layers of network, and l is set to 3, then the first 3 layers of network are intercepted as a shallow sub-network, a lightweight adapter is connected to the output end, and the language modeling head of the model originally used to generate text results is connected to the output end of the lightweight adapter, thus constructing a self-drafting model for preliminary generation of candidate text sequences.

[0022] In some possible embodiments, in order to make the adapter have sufficient representation conversion capability while keeping a very low parameter size, the lightweight adapter can be composed of a multi-head attention module and two normalization layers connected in sequence; the multi-head attention module is used to aggregate context information across semantic units for the features output by the shallow sub-network, in order to capture necessary sequence dependencies; the two normalization layers are connected at the input and output ends of the multi-head attention module, respectively, to stabilize the feature distribution during training and inference, and prevent numerical instability.

[0023] Exemplarily, in order to enable the self-drafting model to effectively imitate the output behavior of the preset decoding main model, control its output results to be closer to the preset decoding main model, and thus obtain a higher candidate acceptance rate in the inference stage, after the self-drafting model is constructed, the distribution alignment training can also be performed on it, with reference to Figure 3 As shown in the figure, the specific training process can include the following steps S301-S304: S301, freeze the model parameters of the preset decoding main model, and obtain a training data set.

[0024] Here, in the training process of the self-drafting model, all parameters of the preset decoding main model are locked and do not participate in subsequent training updates, so as to ensure that its original language modeling ability remains unchanged.

[0025] Specifically, the training data set can include a large-scale text corpus, such as processed dialogue data (such as ShareGPT format data) or general text corpus, which is organized into multiple training data subsets (such as multiple training batches), and each training data subset contains multiple training sample pairs. Each training sample pair is composed of a sequence prefix and its corresponding next semantic unit. The sequence prefix refers to a text sequence, which is used to simulate the historical context in the autoregressive generation process; the next semantic unit is the real word unit (i.e., token) that appears after the sequence prefix, which provides an explicit local prediction target for training. For example, for the sentence "Artificial intelligence is changing the world", a sequence prefix "Artificial intelligence is" and a next semantic unit "change" can be generated as a training sample pair.

[0026] Exemplarily, the training data set can also be constructed and adapted according to specific application fields to enhance the drafting quality of the self-drafting model in specific scenarios. For example, in the financial service field, its training data set can be constructed based on professional texts such as financial research reports, listed company announcements, and financial news, and after cleaning and formatting, sequence samples are formed for training the model to generate texts that conform to the financial context and specifications; or in the medical and health field, its training data set can be derived from desensitized clinical medical records, medical literature abstracts, or medical device instruction manuals, etc., aiming to enable the model to master medical terminology and report writing logic. Here, the specific composition, source field and preprocessing method of the training data set are not specifically limited, and those skilled in the art can reasonably design and prepare them according to the target task and available data resources.

[0027] S302, input the sequence prefix corresponding to each training sample pair in each training data subset into the self-drafting model to obtain a first prediction probability distribution output by the self-drafting model for the next semantic unit corresponding to the training sample pair; and input the sequence prefix corresponding to the training sample pair into the preset decoding main model to obtain a second prediction probability distribution output by the preset decoding main model for the next semantic unit corresponding to the training sample pair.

[0028] Here, for each training sample pair in each training data subset, double forward calculation is performed to obtain the output probability distribution of the self-drafting model and the preset decoding main model respectively in parallel, so as to realize the quantification of the difference between the two in the next step. This process can be embodied as: for the same input context, the prediction results of the student model (self-drafting model) and the teacher model (preset decoding main model) for the same correct target are obtained in parallel, which provides direct comparison data for subsequent distribution difference measurement.

[0029] Specifically, the sequence prefix corresponding to the current training sample pair is input into the self-drafting model to be trained, the self-drafting model processes the prefix based on its current parameters (mainly the parameters of the lightweight adapter), and outputs the prediction probability value of the specified next semantic unit in the sample pair, which constitutes the scalar embodiment of the first prediction probability distribution on this target unit (the distribution can be obtained by combining the softmax function output of the entire vocabulary). For example, for the sequence prefix "today weather is very", the self-drafting model may output a probability of 0.6 for "good", a probability of 0.3 for "bad", and a probability of 0.1 for "sunny", etc. At the same time, the same sequence prefix is input into the preset decoding main model which has been frozen, the main model performs calculation based on its complete and fixed parameters, and outputs the prediction probability value of the same next semantic unit, which constitutes the scalar embodiment of the second prediction probability distribution on this target unit. Again taking "today weather is very" as an example, the preset decoding main model may output a probability of 0.7 for "good", a probability of 0.2 for "bad", and a probability of 0.1 for "sunny", etc. Through the above steps, two different prediction probability distributions of the self-drafting model and the preset decoding main model for the same sequence prefix and next semantic unit can be obtained.

[0030] S303, based on the first prediction probability distribution and the second prediction probability distribution corresponding to each training sample pair in each training data subset, calculate the difference between the self-drafting model and the preset decoding main model to determine the model alignment loss.

[0031] Specifically, after obtaining the corresponding first and second prediction probability distributions of each training sample in each training data subset, the inconsistency of the two model output distributions can be quantified using a loss function. Here, the cross-entropy loss function or the KL divergence can be used. When the cross-entropy loss function is used, the cross-entropy between the first prediction probability distribution output by the self-drafting model and the second prediction probability distribution output by the preset decoding main model (or the probability value of the distribution for the target next semantic unit) is calculated as a "soft label" or supervision signal for each training sample. Then, the loss values of all samples in a training data subset (batch) are averaged or summed to obtain the total model alignment loss of the batch, which reflects the error of the self-drafting model in imitating the output distribution of the preset decoding main model.

[0032] S304, updating the model parameters of the self-drafting model according to the model alignment loss, and returning to step S302 based on the updated self-drafting model until a preset alignment training condition is met, obtaining a self-drafting model aligned with the output distribution of the preset decoding main model.

[0033] It can be understood that the purpose of updating the model parameters is to make the output of the self-drafting model closer to the preset decoding main model by adjusting the internal parameters of the self-drafting model, thereby reducing the model alignment loss. Here, the backpropagation algorithm can be used to calculate the gradient of the loss function with respect to the trainable parameters of the self-drafting model (i.e., all weights and biases of the lightweight adapter), and then use an optimizer (e.g., AdamW optimizer) to update these parameters according to the calculated gradient. After updating the model parameters, the above step S302 can be returned based on the updated self-drafting model, and the input, prediction probability distribution calculation, etc. operations are performed again. This process will be repeated continuously until the preset alignment training condition is met. The preset alignment training condition can be that the model alignment loss converges to a stable value on the validation set, the training reaches a preset number of epochs, or the model no longer improves on the specific indicators of the reserved validation set, etc.

[0034] Further, when the training is terminated, the output probability distribution of the obtained self-drafting model can well approximate the distribution of the preset decoding main model, thereby laying a foundation for efficient and high-acceptance self-drafting in the inference stage. The entire training process only updates the parameters of the lightweight adapter, and the preset decoding main model remains frozen, greatly reducing the training overhead and avoiding damaging the original performance of the main model.

[0035] S202, obtaining a to-be-decoded sequence, and inputting the to-be-decoded sequence to the preset decoding main model, performing first heavy hierarchical early exit when calculating to the l-th layer, obtaining the hidden state output by the l-th layer; and inputting the hidden state to the self-drafting model.

[0036] Here, the to-be-decoded sequence refers to the historical context that currently needs to continue to generate text, for example, an unfinished dialogue, for example, in an unfinished dialogue with an intelligent customer service, the sequence can contain the user's historical question "how to upgrade insurance?" and the system's generated reply beginning "please first open the insurance application software, then"; or in a code generation scenario, it can be an existing function definition and annotation. Input the sequence into the preset decoding main model, and start the forward calculation process of the preset decoding main model (which can also be understood as the shallow subnetwork of the self-drafting model). When the calculation process proceeds to the l-th layer, that is, the last layer of the shallow subnetwork, the subsequent deeper layer (l+1-th layer to the last layer) calculation of the main model is actively terminated. This operation is the first heavy level early exit. Subsequently, the hidden state output at the l-th layer at this moment is obtained, which is a high-dimensional vector encoding the semantic information of the current context after shallow understanding, and finally the hidden state can be provided as input to the self-drafting model constructed above.

[0037] In this way, instead of running the complete external draft model in the traditional speculative decoding process, the drafting starting point is efficiently and low-costly obtained from the inside of the main model.

[0038] S203, based on the input hidden state, performing multi-step iteration by the self-drafting model to generate a candidate semantic unit sequence; during the multi-step iteration process, if the highest probability of the next semantic unit predicted by the current iteration step is lower than a preset confidence threshold, performing second heavy semantic unit level early exit to stop iteration.

[0039] It can be understood that the self-drafting model takes the hidden state obtained in the last step as the starting point, performs multi-step iteration in an autoregressive manner, and gradually predicts a candidate semantic unit sequence. Among them, the semantic unit is only the basic unit generated by the language model, such as a word or a subword. In each iteration step, the self-drafting model will output the probability distribution of the next semantic unit. If the probability value of the semantic unit with the highest probability predicted by the current iteration step is lower than a preset confidence threshold, the second heavy semantic unit level early exit is triggered to stop subsequent iteration. For example, in the process of generating the sequence "cat sat on the", if the probability of the candidate word with the highest probability is only 0.3 when predicting the next unit, which is lower than the threshold 0.65, the generation is stopped. In this way, it is ensured that the drafting process will not waste calculation in the low confidence area, and the length of the current candidate sequence is dynamically determined.

[0040] Exemplarily, with reference to FIG. 4, Figure 4 The process of generating a candidate semantic unit sequence by multi-step iteration of the self-drafting model can include the following steps S401-S404: S401, generate a first candidate semantic unit by a backend part of the self-drafting model, taking the input hidden state as an initial state.

[0041] Specifically, the self-drafting model is divided into two parts, i.e., a front-end part and a back-end part. The front-end part includes a shallow sub-network, which is responsible for receiving a text sequence and extracting a corresponding hidden state representation thereof. The back-end part includes a light adapter and a language modeling head, which are responsible for converting the hidden state into a probability distribution over a vocabulary and sampling to generate a specific semantic unit.

[0042] Here, the hidden state obtained by early exiting from the first heavy level is directly input to the back-end part of the self-drafting model, bypassing the repeated calculation of the front-end part of the self-drafting model. The back-end part can output a first candidate semantic unit based on the state. For example, in the generation task of a financial analysis report, assuming that the decoding sequence is "the risk assessment of the portfolio shows", the hidden state obtained from the shallow layer of the main model encodes the preliminary semantics of this context, and the back-end part of the self-drafting model generates the first candidate word "insurance" based on this.

[0043] S402, concatenate the first candidate semantic unit to the decoding sequence to form an initial intermediate sequence; input the initial intermediate sequence into the front-end part of the self-drafting model to obtain a new hidden state, and input the new hidden state into the back-end part of the self-drafting model to generate a next candidate semantic unit.

[0044] It can be understood that the initial intermediate sequence is formed by connecting the complete decoding text and the first candidate semantic unit, which represents the entire context generated so far. Compared with the original decoding sequence, it is more content-rich, providing more context information for generating more accurate candidate semantic units. Further, the sequence is sent to the front-end part of the self-drafting model, i.e., the shallow sub-network, and a new hidden state vector is output after the calculation of the first l layers. This new hidden state vector integrates more semantic features of the initial intermediate sequence and is more comprehensive and accurate than the initial hidden state. Then, the vector is sent to the back-end part, and a next candidate semantic unit is generated based on the new hidden state after the transformation by the light adapter and the mapping by the language modeling head.

[0045] Here, continuing the example, the first candidate "insurance" is concatenated to the decoding sequence to form a new initial intermediate sequence "the risk assessment of the portfolio shows insurance". The sequence is input to the front-end part of the self-drafting model (shallow sub-network), and a new hidden state is output after the calculation, which integrates the updated semantics after the addition of the word "insurance". The back-end part can generate a next candidate semantic unit, e.g., "product", based on the new state.

[0046] S403, repeating the above steps for iteration, in each iteration, concatenating the candidate semantic unit generated in this iteration to the intermediate sequence formed in the last iteration to form a new intermediate sequence, and determining a new hidden state based on the new intermediate sequence through the front-end part of the self-drafting model until a preset stopping condition is met and / or the second heavy semantic unit level early exit is triggered, arranging all candidate semantic units generated in the iteration process in the order of generation to form the candidate semantic unit sequence.

[0047] It can be understood that the iteration process is a typical autoregressive generation loop, each loop includes: context sequence update (concatenating the newly generated semantic unit), forward feature extraction (obtaining a new hidden state through the front-end part), and next unit prediction (generating a new candidate through the back-end part). In each iteration, it is checked whether the preset stopping condition is met and / or whether the second heavy semantic unit level early exit is triggered. When the loop is terminated due to meeting the stopping condition, the candidates generated in each step are collected in sequence to constitute a complete candidate semantic unit sequence.

[0048] The preset stopping condition can include jumping out of the iteration after reaching the preset maximum number of steps of a single round of drafting, or other conditions set according to task requirements or resource constraints, such as reaching a preset maximum calculation time, observing that the confidence of the generated semantic unit presents a downward trend for a plurality of consecutive times, or the total calculation amount of the current batch reaches a threshold, etc., which is not limited here.

[0049] Exemplarily, continuing the above example, after the second iteration, the intermediate sequence is updated to "the portfolio's risk assessment shows - insurance - product", which is input into the front-end part to obtain a new hidden state, and the back-end part can further generate a candidate "with". This iteration process continues, for example, the candidates "high", "return", and "rate" can be generated in turn. The stopping condition can be triggered due to meeting the preset number of steps (such as a maximum of 6 words), or the second early exit can be triggered due to the highest probability value of a certain word (such as "of") being lower than a preset confidence threshold when generating the word. Assuming that the iteration is stopped after generating the sequence "insurance - product - with - high - return - rate" due to reaching the step limit, then the sequence is the candidate semantic unit sequence generated this time.

[0050] S404, recording the intermediate hidden state output by the front-end part of the self-drafting model in each iteration step.

[0051] Specifically, in the drafting iteration process, the new hidden state is calculated by the front-end part (shallow sub-network) at each pass, and the state vector is saved. These intermediate hidden states saved in the generation order correspond to each position in the candidate semantic unit sequence, and together form the input of the subsequent parallel verification stage. For example, when generating the candidate "insurance", the state H1 is recorded; when generating "product" based on "insurance", the state H2 is recorded; and when generating "has", "high", "return", and "rate" subsequently, the states H3, H4, H5, and H6 are recorded respectively. Finally, the six intermediate hidden states [H1, H2, H3, H4, H5, H6] recorded in order correspond to the six candidate semantic units and will be used for subsequent parallel verification. In this way, recording these states avoids re-computing the shallow layer in the verification stage to achieve subsequent computation reuse.

[0052] In the embodiments of the present disclosure, the drafting process is decoupled from the shallow calculation of the main model through structural design, avoiding repeated forward transmission of the same context, and recording all intermediate hidden states for subsequent verification stage reuse. At the same time, the dynamic control of the candidate sequence length is realized through the confidence threshold in the iteration, achieving adaptive balance between generation quality and drafting efficiency, effectively reducing the computational overhead and delay of a single round of drafting, and providing complete intermediate states that can be directly used for subsequent parallel verification, avoiding repeated shallow calculation in the verification stage.

[0053] S204, splicing the intermediate hidden state output by the shallow sub-network corresponding to each iteration step in the self-drafting model multi-step iteration process to form a parallel tensor; inputting the parallel tensor to the remaining network layer after the preset decoding main model lth layer for parallel forward calculation to obtain the verification result of each position in the candidate semantic unit sequence.

[0054] Specifically, in each step of the self-drafting model iteration to generate a candidate sequence, the shallow sub-network will correspondingly output an intermediate hidden state. The intermediate hidden state refers to the high-dimensional feature vector output by the last layer (lth layer) of the shallow sub-network after inputting the complete intermediate sequence generated up to the current iteration step into the shallow sub-network, which encodes the semantic information of the context at the specific sequence position understood by the shallow layer of the main model. At this time, these intermediate hidden states corresponding to all candidate positions are spliced in the batch dimension to form a parallel tensor. The tensor is a multidimensional data array in deep learning, for example, a two-dimensional floating-point matrix with a shape of [γ, d], where γ represents the number of candidate semantic units, and d represents the dimension of the hidden state. If 6 candidates are generated this time and the hidden dimension is 768, the parallel tensor is a [6, 768] matrix.

[0055] Further, the parallel tensor can be input to all remaining network layers after the lth layer of the preset decoding main model, and the remaining layers perform parallel forward calculation on the batch input, and output the verification results of all candidate semantic unit positions at one time, i.e., the determination of the semantic unit that should be generated for each position by the main model based on complete deep calculation.

[0056] Here, in order to clearly describe the complete process of verifying a plurality of candidate intermediate representations through the deep network of the main model at one time, referring to FIG. 5, the parallel verification process can include the following steps S501-S504: Figure 5 S501, input the parallel tensor to the (l+1)th layer of the preset decoding main model.

[0057] Here, the parallel tensor contains the intermediate hidden states generated for a plurality of candidate semantic units from the drafting stage, and the states are batch input to the first deep layer, i.e., the (l+1)th layer, which is the starting point of the deep feature extraction of the main model, after the shallow sub-network, and receives all candidate context representations at the same time in batch processing mode.

[0058] S502, sequentially process the parallel tensor through the network of the (l+1)th layer to the last layer of the preset decoding main model, and output the final feature vector corresponding to each candidate semantic unit in the candidate semantic unit sequence.

[0059] Specifically, starting from the (l+1)th layer, the parallel tensor will pass through all the remaining deep network layers (e.g., the (l+2)th, (l+3)th, …, Lth layer) of the main model in turn, and each layer performs its inherent calculation operation, such as multi-head attention mechanism and feedforward neural network, on the batch input, which is mathematically applied to each sample in the batch independently, but the calculation is performed in parallel on hardware. After deep and complex nonlinear transformation, the original intermediate hidden state is refined and deepened layer by layer, and finally a set of final feature vectors is output at the Lth layer (the last layer). The set of vectors is consistent with the input parallel tensor in the batch dimension, i.e., each candidate semantic unit obtains a unique high-level semantic representation fully understood through the complete deep network of the main model.

[0060] S503, input the final feature vector to the language modeling head of the preset decoding main model to obtain the probability distribution of each candidate semantic unit in the candidate semantic unit sequence by the preset decoding main model.

[0061] ​Here, the language modeling head is a linear layer of the main model, whose weight matrix maps the dimension of the final feature vector to the size of the entire vocabulary. Each final feature vector is independently passed through this language modeling head, linearly transformed, and then normalized by a Softmax function, to generate a probability distribution over all possible words for the candidate position corresponding to this vector. This distribution represents the preset decoding main model, which, based on all its parameters and full depth calculation, determines the complete answer to the question of which semantic unit should appear in this particular position, given the historical context and all previously accepted candidates. For example, in the medical report generation scenario, assuming the candidate semantic unit sequence is [pneumonia, patient, needs, hospitalization, observation], the probability distribution generated by the main model for the first position "pneumonia" may show that the probability of "pneumonia" is 0.85, the probability of "infection" is 0.10, and the probability of "asthma" is 0.05; the probability distribution generated for the second position "patient" may show that the probability of "patient" is 0.90, the probability of "patient" is 0.08, and the probability of "this" is 0.02.

[0062] S504, for each candidate semantic unit in the candidate semantic unit sequence, selecting the semantic unit with the highest probability value from the probability distribution corresponding to the candidate semantic unit as the verification result of the candidate semantic unit.

[0063] Specifically, after the probability distribution of each position is calculated, a maximum probability selection operation can be performed, that is, the semantic unit with the highest probability value is found, which is the official judgment or verification result of the preset decoding main model for the candidate position. For example, in combination with the above example, for the first candidate position "pneumonia", the "pneumonia" with the highest probability is selected from its distribution as the verification result; for the second candidate position "patient", the "patient" with the highest probability is selected from its distribution as the verification result. If in the financial risk prompt scenario, the candidate sequence is [market, volatility, intensifies], and the verification result of the main model for the second position is "risk" (probability 0.6), which is higher than the candidate "volatility" (probability 0.35), then the verification result of this position is "risk".

[0064] S205, determining the target decoding sequence according to the verification result and the order comparison result of the candidate semantic unit sequence.

[0065] Here, the main model verification result obtained in step S204 is sequentially compared with the candidate semantic unit sequence generated from the draft model in step S203, and the semantic units considered reasonable in the verification result can be screened out and combined into a target decoding sequence according to the order of the candidate semantic unit sequence. The target decoding sequence is the final decoding result. For example, in the medical report generation scenario, if the candidate sequence is [pneumonia, patient, need, hospitalization, observation], and the verification result sequence is [pneumonia, patient, need, treatment, follow-up], after sequential comparison, the first three semantic units “pneumonia”, “patient” and “need” are continuously matched, and the fourth position “hospitalization” and “treatment” are not matched. Therefore, the first three semantic units “pneumonia”, “patient” and “need” accepted in this round are added to the historical context to form a target decoding sequence, which can be used as the new input of the next decoding process. For another example, in the financial report generation, if the candidate sequence is [market, fluctuation, intensification], and the verification result sequence is [market, fluctuation, intensification], the two are completely consistent, and all three candidates are accepted to form the target decoding sequence.

[0066] For example, to ensure that the decoding process strictly follows the main reference determination of the main model and keeps the output lossless, referring to FIG. 6, when determining the target decoding sequence as the final output according to the sequential comparison between the verification result sequence and the candidate semantic unit sequence, the following steps S601-S604 can be included: Figure 6 S601, comparing the semantic unit in the verification result with the corresponding semantic unit in the candidate semantic unit sequence from the first position of the candidate semantic unit sequence.

[0067] Here, a sequential comparison process can be started to check the authoritative verification result of the main model and the candidate sequence proposed by the draft model element by element. Specifically, a position index can be established and maintained, that is, a position pointer (for example, index i=0) is initialized, and a loop process is entered, in each round of loop, the semantic unit at the i-th position in the verification result sequence is taken out, and the semantic unit at the i-th position in the candidate semantic unit sequence is taken out. The identity of the two is judged. The comparison basis can be the complete match of the string or identifier, to confirm whether the prediction of the draft model at the current position is completely consistent with the final determination obtained by the main model based on the complete depth. For example, in the financial text generation, if the first position of the candidate sequence is “stock price”, and the first position of the verification result is also “stock price”, it is determined to be matched; if the candidate is “rise” and the verification result is “yield”, it is determined to be not matched, and the comparison process will be terminated at this position.

[0068] S602, accepting the continuously matched semantic units from the starting position of the candidate semantic unit sequence.

[0069] ​Specifically, if the comparison result at the current position is a match, the candidate semantic unit can be marked as "accepted" and temporarily stored in an accepted list; then, the position pointer is incremented (i = i + 1), and the loop returns to step S601 to continue comparing the next position. This process continues until a mismatch occurs at a position or all positions are compared.

[0070] In this way, it is ensured that the prefix part of the output sequence is absolutely consistent with the judgment of the main model, and the quality of the generated result is guaranteed to be the same as the baseline model distribution.

[0071] S603, if the first mismatched semantic unit occurs, the first mismatched semantic unit and all candidate semantic units after it are deleted.

[0072] Specifically, due to the characteristics of autoregressive generation: the occurrence of the first error position will destroy the context basis of all subsequent predictions, and the subsequent candidates are theoretically unreliable. Therefore, when the semantic unit of the verification result is different from the candidate semantic unit at a certain position k, according to the lossless rule of the standard of speculative decoding, the current drafting round is immediately terminated. At this time, the position k and all candidate semantic units after it (i.e., the part with an index greater than or equal to k in the candidate sequence) are discarded as a whole, regardless of whether these subsequent candidates themselves match the verification result. For example, in medical text generation, if a mismatch occurs at the third position, even if the fourth candidate matches, the fourth, fifth, and subsequent candidates are also deleted.

[0073] S604, all accepted semantic units are spliced to the end of the to-be-decoded sequence to obtain the target decoding sequence.

[0074] Further, after the drafting round is terminated, all accepted semantic units can be taken out in the original order and appended to the tail of the to-be-decoded sequence input at the beginning of the current decoding round as a continuous segment. The new text sequence generated by splicing is the target decoding sequence of the current round, which represents the final confirmed output of the model up to the current time. This sequence can be used as the input (i.e., the new to-be-decoded sequence) of the next decoding, thereby driving the generation process to iterate forward.

[0075] In this way, by sequential comparison and the prefix matching principle, the acceptance is stopped only after the continuous prefix segment in which the main model verification result and the self-drafted candidate are completely consistent, ensuring that the probability distribution of the output sequence is consistent with the autoregressive decoding of the original main model, and realizing the incremental and lossless generation of the text.

[0076] It can be understood that the above single round decoding process will be iteratively executed until the generation task ends (such as a specific end marker is generated, the output sequence reaches a preset maximum length, or a specific business logic completion condition is met). In order to make full use of the effective intermediate calculation results completed in the self-drafting stage, reduce unnecessary repeated calculation between decoding rounds, and further improve the overall inference efficiency, after determining the target decoding sequence, the following steps (1)~(2) can also be included: (1) record the intermediate hidden state corresponding to the last accepted candidate semantic unit; (2) in the next decoding round, the recorded intermediate hidden state corresponding to the last accepted candidate semantic unit is taken as the initial state of the first candidate semantic unit generated by the self-drafting model to start a new round of self-drafting process.

[0077] Here, no matter how many semantic units in the current round of candidate sequence are accepted, as long as there is at least one accepted unit, the intermediate hidden state output by the front-end part (shallow subnetwork) of the self-drafting model when generating the last accepted unit can be recorded. The state vector is calculated based on the complete sequence context up to the accepted unit and contains rich semantic information.

[0078] Further, when entering the next round of decoding and preparing to perform the self-drafting step, the process of "inputting the to-be-decoded sequence into the preset decoding main model and calculating to the 1st layer to obtain a new hidden state" in the conventional process can be bypassed or simplified, and the pre-recorded intermediate hidden state from the end of the last round can be directly provided to the back-end part (lightweight adapter and language modeling head) of the self-drafting model as the starting basis for generating the first candidate semantic unit in the new round.

[0079] In this way, the self-drafting model is provided with a "hot start" state, which can start work based on a representation highly related to the latest context and fully calculated, significantly reducing the computational overhead and delay caused by re-computing shallow features between rounds, and realizing efficient connection and optimized reuse of computational amount between decoding processes.

[0080] The self-speculation decoding method, device, medium and equipment based on double early exit provided in the embodiments of the present disclosure realize self-speculation decoding without independent draft model and with dynamic control of draft length, significantly reduce deployment and computational overhead, and improve end-to-end inference efficiency on the premise of ensuring generation quality and consistent distribution with the baseline model, through the cooperation of the above double early exit mechanism and parallel computing strategy, and the self-drafting model composed of the shallow subnetwork of the main model and the lightweight adapter.

[0081] Those skilled in the art can understand that, in the above method of the specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process, and the specific execution order of each step should be determined by its function and possible internal logic.

[0082] Based on the same inventive concept, the disclosure embodiments also provide a double-early-exit-based self-speculation decoding device corresponding to the double-early-exit-based self-speculation decoding method. Since the principle of solving problems in the device of the disclosure embodiments is similar to the above-mentioned double-early-exit-based self-speculation decoding method of the disclosure embodiments, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0083] Referring to Figure 7 Fig. 7 shows a schematic diagram of a double-early-exit-based self-speculation decoding device 700 provided by the disclosure embodiments, the device comprises: A model construction module 701 is configured to intercept the first l layers of a preset decoding main model as a shallow subnetwork, connect a lightweight adapter to the output end of the shallow subnetwork, and connect a language modeling head of the preset decoding main model to the output end of the lightweight adapter, to construct a self-drafting model. A drafting triggering module 702 is configured to obtain a to-be-decoded sequence, input the to-be-decoded sequence into the preset decoding main model, perform a first heavy layer early exit when computing to the first l layer, obtain the hidden state output by the first l layer, and input the hidden state into the self-drafting model. A semantic drafting module 703 is configured to generate a candidate semantic unit sequence by the self-drafting model based on the input hidden state through multi-step iteration, and perform a second heavy semantic unit early exit and stop iteration if the highest probability of the next semantic unit predicted by the current iteration step is lower than a preset confidence threshold. A parallel verification module 704 is configured to splice the intermediate hidden state output by the shallow subnetwork corresponding to each iteration step in the multi-step iteration process of the self-drafting model to form a parallel tensor, input the parallel tensor into the remaining network layers after the first l layer of the preset decoding main model, and perform parallel forward calculation to obtain a verification result for each position in the candidate semantic unit sequence. A sequence determination module 705 is configured to determine a target decoding sequence according to the verification result and the order comparison result of the candidate semantic unit sequence.

[0084] In some possible embodiments, the model construction module 701 is further configured to perform: Step a, freezing the model parameters of the preset decoding main model, and obtaining a training data set; wherein the training data set comprises a plurality of training data subsets, and each training data subset comprises a plurality of training sample pairs composed of a sequence prefix and a corresponding next semantic unit; Step b, for each training sample pair in each training data subset, inputting the sequence prefix corresponding to the training sample pair into the self-drafting model to obtain a first prediction probability distribution output by the self-drafting model for the next semantic unit corresponding to the training sample pair; and inputting the sequence prefix corresponding to the training sample pair into the preset decoding main model to obtain a second prediction probability distribution output by the preset decoding main model for the next semantic unit corresponding to the training sample pair; Step c, based on the first prediction probability distribution and the second prediction probability distribution corresponding to each training sample pair in each training data subset, calculating the difference between the self-drafting model and the preset decoding main model to determine a model alignment loss; Step d, updating the model parameters of the self-drafting model according to the model alignment loss, and returning to step b based on the updated self-drafting model until a preset alignment training condition is met, to obtain a self-drafting model aligned with the output distribution of the preset decoding main model.

[0085] In some possible embodiments, the semantic drafting module 703 is specifically configured to: generate a first candidate semantic unit through the back-end part of the self-drafting model with the input hidden state as an initial state; wherein the self-drafting model comprises a front-end part and a back-end part, the front-end part comprises the shallow sub-network, and the back-end part comprises the light adapter and the language modeling head; splice the first candidate semantic unit to the to-be-decoded sequence to form an initial intermediate sequence, input the initial intermediate sequence into the front-end part of the self-drafting model to obtain a new hidden state, and input the new hidden state into the back-end part of the self-drafting model to generate a next candidate semantic unit; repeat the above steps for iteration, in each iteration, splice the candidate semantic unit generated at this time to the intermediate sequence formed in the last iteration to form a new intermediate sequence, and determine a new hidden state based on the new intermediate sequence through the front-end part of the self-drafting model, until a preset stop condition is met and / or the second semantic unit level early exit is triggered, arrange all candidate semantic units generated in the iteration process in the order of generation to form the candidate semantic unit sequence; record the intermediate hidden state output by the front-end part of the self-drafting model in each iteration step.

[0086] In some possible embodiments, the parallel verification module 704 is specifically configured to: input the parallel tensor into an (l+1)th layer of the preset decoding main model; sequentially process the parallel tensor through a network of the (l+1)th layer to a last layer of the preset decoding main model, and output a final feature vector corresponding to each candidate semantic unit in the candidate semantic unit sequence; input the final feature vector into a language modeling head of the preset decoding main model, to obtain a probability distribution of each candidate semantic unit in the candidate semantic unit sequence by the preset decoding main model; for each candidate semantic unit in the candidate semantic unit sequence, select a semantic unit with a highest probability value from the probability distribution corresponding to the candidate semantic unit as a verification result of the candidate semantic unit.

[0087] In some possible embodiments, the sequence determination module 705 is specifically configured to: start from a first position of the candidate semantic unit sequence, compare the semantic unit in the verification result with a corresponding semantic unit in the candidate semantic unit sequence; accept the continuously matched semantic units starting from the first position of the candidate semantic unit sequence; if a first unmatched semantic unit appears, delete the first unmatched semantic unit and all candidate semantic units after the first unmatched semantic unit; splice all the accepted semantic units to the end of the to-be-decoded sequence to obtain the target decoding sequence.

[0088] In some possible embodiments, the sequence determination module 705 is further configured to: record an intermediate hidden state corresponding to the last accepted candidate semantic unit; in a next decoding round, take the recorded intermediate hidden state corresponding to the last accepted candidate semantic unit as an initial state of the first candidate semantic unit generated by the self-crafting model, to start a new round of self-crafting process.

[0089] In some possible embodiments, the lightweight adapter is sequentially connected by one multi-head attention module and two normalization layers; the multi-head attention module is configured to aggregate cross-semantic-unit context information of the features output by the shallow subnetwork; the two normalization layers are respectively connected to input and output ends of the multi-head attention module, and are configured to stabilize feature distribution in a training and inference process.

[0090] Based on the same technical concept, the embodiments of the present disclosure further provide a computer device. Referring to Figure 8 As shown in FIG. 8, a structural schematic diagram of a computer device 800 provided by the embodiments of the present disclosure includes a processor 801, a memory 802, and a bus 803. The memory 802 is configured to store execution instructions, including an internal memory 8021 and an external memory 8022. The internal memory 8021 is also referred to as an internal storage, and is configured to temporarily store operation data in the processor 801 and exchange data with the external memory 8022 such as a hard disk. The processor 801 exchanges data with the external memory 8022 through the internal memory 8021.

[0091] In the embodiments of the present application, the memory 802 is specifically configured to store application program codes for executing the schemes of the present application, and is controlled by the processor 801 to execute. That is, when the computer device 800 is running, the processor 801 and the memory 802 communicate through the bus 803, so that the processor 801 executes the application program codes stored in the memory 802, and further executes the methods described in any of the preceding embodiments.

[0092] The memory 802 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and the like.

[0093] The processor 801 can be an integrated circuit chip having a signal processing capability. The processor described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), or the like; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed by the processor. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like.

[0094] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the computer device 800. In some other embodiments of the present application, the computer device 800 can include more or fewer components than those illustrated, or combine certain components, or split certain components, or different arrangement of components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0095] The embodiments of the present disclosure also provide a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the self-speculation decoding method based on double early exit described in the above method embodiments are executed. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0096] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the self-speculation decoding method based on double early exit described in the above method embodiments. For details, refer to the above method embodiments, which will not be described here.

[0097] The computer program product can be specifically implemented by hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) or the like.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here. In several embodiments provided in the present disclosure, it should be understood that the disclosed system and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and there can be another division in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0099] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0100] In addition, the functional units in each embodiment of the present disclosure can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0101] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part of the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the various embodiments of the present disclosure. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, and various program codes that can be stored in the medium.

[0102] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and are not intended to limit the present disclosure. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easy changes to the technical solutions described in the foregoing embodiments, or easily think of changes or equivalent replacements for some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A self-predictive decoding method based on double early exit, characterized in that, include: The first l layers of the network in the preset decoding master model are extracted as shallow sub-networks, and a lightweight adapter is connected to the output of the shallow sub-network. The language modeling head of the preset decoding master model is connected to the output of the lightweight adapter to construct a self-drafting model. Obtain the sequence to be decoded and input the sequence to be decoded into the preset decoding master model. When the calculation reaches the l-th layer, perform the first level early exit and obtain the hidden state output by the l-th layer. The hidden state is then input into the self-drafting model; Based on the input hidden state, the self-drafting model performs multiple iterations to generate a sequence of candidate semantic units. During the multi-step iteration process, if the highest probability of the next semantic unit predicted in the current iteration step is lower than the preset confidence threshold, the second semantic unit level is executed to exit early and the iteration stops. The intermediate hidden states output by the shallow sub-network corresponding to each iteration step in the multi-step iteration process of the self-drafting model are concatenated to form a parallel tensor; the parallel tensor is input to the remaining network layers after the l-th layer of the preset decoding main model to perform parallel forward computation and obtain the verification result for each position in the candidate semantic unit sequence. The target decoding sequence is determined based on the alignment results of the verification results and the candidate semantic unit sequences.

2. The method according to claim 1, characterized in that, The construction of the self-drafted model includes: Step a: Freeze the model parameters of the preset decoding master model and obtain the training dataset; wherein, the training dataset includes multiple training data subsets, and each training data subset includes multiple training sample pairs consisting of a sequence prefix and the corresponding next semantic unit; Step b: For each training sample pair in each training data subset, input the sequence prefix corresponding to the training sample pair into the self-drafting model to obtain a first prediction probability distribution output by the self-drafting model for the next semantic unit corresponding to the training sample pair; and input the sequence prefix corresponding to the training sample pair into the preset decoding master model to obtain a second prediction probability distribution output by the preset decoding master model for the next semantic unit corresponding to the training sample pair. Step c: Based on the first and second prediction probability distributions corresponding to each training sample pair in each training data subset, calculate the difference between the self-drafting model and the preset decoding master model, and determine the model alignment loss; Step d: Update the model parameters of the self-drafting model according to the model alignment loss, and return to step b above based on the updated self-drafting model until the preset alignment training conditions are met, so as to obtain a self-drafting model that is aligned with the output distribution of the preset decoding master model.

3. The method according to claim 1, characterized in that, The step of generating a sequence of candidate semantic units through multi-step iterations of the self-drafting model includes: Using the input hidden state as the initial state, the first candidate semantic unit is generated through the back-end part of the self-drafting model; wherein, the self-drafting model includes a front-end part and a back-end part, the front-end part includes the shallow sub-network, and the back-end part includes the lightweight adapter and the language modeling head; The first candidate semantic unit is concatenated to the sequence to be decoded to form an initial intermediate sequence; the initial intermediate sequence is input into the front end of the self-drafting model to obtain a new hidden state, and the new hidden state is input into the back end of the self-drafting model to generate the next candidate semantic unit. Repeat the above steps for iteration. In each iteration, the candidate semantic units generated in the current iteration are concatenated to the intermediate sequence formed in the previous iteration to form a new intermediate sequence. Based on the new intermediate sequence, a new hidden state is determined through the front end of the self-drafting model until the preset stopping condition is met and / or the second semantic unit level is triggered to exit early. All candidate semantic units generated during the iteration process are arranged in the order of generation to form the candidate semantic unit sequence. Record the intermediate hidden states output by the front end of the self-drafting model in each iteration step.

4. The method according to claim 3, characterized in that, The step of inputting the parallel tensor into the remaining network layers after the l-th layer of the preset decoding master model for parallel forward computation includes: The parallel tensor is input into the (l+1)th layer of the preset decoding master model; The parallel tensor is processed sequentially by the network from the (l+1)th layer to the last layer of the preset decoding master model, and the final feature vector corresponding to each candidate semantic unit in the candidate semantic unit sequence is output. The final feature vector is input into the language modeling head of the preset decoding master model to obtain the probability distribution of each candidate semantic unit in the candidate semantic unit sequence by the preset decoding master model; For each candidate semantic unit in the candidate semantic unit sequence, the semantic unit with the highest probability value is selected from the probability distribution corresponding to the candidate semantic unit as the verification result of the candidate semantic unit.

5. The method according to claim 1, characterized in that, The step of determining the target decoding sequence based on the alignment result of the verification result and the candidate semantic unit sequence includes: Starting from the first position of the candidate semantic unit sequence, the semantic units in the verification result are compared with the corresponding semantic units in the candidate semantic unit sequence; Accept consecutive matching semantic units starting from the beginning position of the candidate semantic unit sequence; If the first mismatched semantic unit is found, then delete the first mismatched semantic unit and all subsequent candidate semantic units; All the received semantic units are concatenated to the end of the sequence to be decoded to obtain the target decoded sequence.

6. The method according to claim 5, characterized in that, After obtaining the target decoded sequence, the process includes: Record the intermediate hidden state corresponding to the last accepted candidate semantic unit; In the next decoding round, the intermediate hidden state corresponding to the last accepted candidate semantic unit is recorded as the initial state for the self-drafting model to generate the first candidate semantic unit, thereby initiating a new round of self-drafting process.

7. The method according to any one of claims 1 to 6, characterized in that, The lightweight adapter consists of a multi-head attention module and two normalization layers connected sequentially. The multi-head attention module is used to aggregate contextual information across semantic units for the features output by the shallow sub-network; The two normalization layers are connected to the input and output of the multi-head attention module, respectively, to stabilize the feature distribution during training and inference.

8. A self-speculating decoding device based on dual early exit, characterized in that, include: The model building module is used to extract the first l layers of the network from the preset decoding master model as shallow sub-networks, connect the lightweight adapter to the output of the shallow sub-networks, and connect the language modeling head of the preset decoding master model to the output of the lightweight adapter to build a self-drafting model. The drafting trigger module is used to obtain the sequence to be decoded and input the sequence to be decoded into the preset decoding master model. When the calculation reaches the l-th layer, the first level of early exit is executed to obtain the hidden state of the output of the l-th layer. The hidden state is then input into the self-drafting model; The semantic drafting module is used to generate a sequence of candidate semantic units by performing multi-step iterations based on the input hidden state using the self-drafting model. During the multi-step iteration process, if the highest probability of the next semantic unit predicted in the current iteration step is lower than a preset confidence threshold, the second semantic unit level is executed to exit early and the iteration is stopped. The parallel verification module is used to concatenate the intermediate hidden states output by the shallow sub-network corresponding to each iteration step in the multi-step iteration process of the self-drafting model to form a parallel tensor; the parallel tensor is input to the remaining network layers after the l-th layer of the preset decoding main model to perform parallel forward computation and obtain the verification result for each position in the candidate semantic unit sequence. The sequence determination module is used to determine the target decoding sequence based on the sequence comparison result between the verification result and the candidate semantic unit sequence.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images