Model speculation decoding method and device based on dynamic layer skipping, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to businesses such as medical health and financial science and technology, and discloses a model speculation decoding method based on dynamic layer skipping, which comprises the following steps: pre-constructing a target large language model, and defining a draft model through the target large language model; obtaining a plurality of layer-skipping schemes, and constructing a layer-skipping mode set according to the layer-skipping schemes; according to the current input state and the layer contribution degree of the target large language model, screening an optimal layer hopping mode; applying the optimal layer-skipping mode to the target large language model, and updating the draft model according to the optimal layer-skipping mode; performing G-step autoregressive reasoning through the updated draft model to obtain a plurality of candidate tokens, and determining the number of acceptable tokens based on the plurality of candidate tokens; starting from the preset starting position, the target large language model carries out text decoding according to the number of the acceptable tokens. According to the method, a training-free draft model is constructed through dynamic layer skipping, the optimal layer skipping is selected through layer contribution degree evaluation, verification candidate tokens are circularly generated, and efficient lossless decoding is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model inference decoding method, apparatus, device and storage medium based on dynamic layer skipping. Background Technology

[0002] Speculative decoding (SD) is an effective means of accelerating inference in large language models (LLMs), but existing technologies still have significant drawbacks and limitations. Traditional speculative decoding methods require training independent small draft models to generate candidate tokens. This process consumes a large amount of corpus and computational resources, resulting in high training costs and impracticality, severely limiting its deployment in real-world scenarios. While existing self-drafting methods such as Medusa rely on the target model itself, they require the introduction of additional parameters such as adapters / headers and need to acquire draft generation capabilities through additional training or fine-tuning, making it difficult to achieve widespread application as a plug-and-play solution. Furthermore, most related models use fixed computational paths and cannot dynamically adjust the amount of computation based on input complexity, contextual redundancy, or the characteristics of the target task, resulting in a large amount of redundant computation during inference, ultimately affecting acceleration efficiency and making it difficult to achieve optimal results.

[0003] In the healthcare field, the shortcomings of existing speculative decoding (SD) technologies directly lead to critical technical issues that contradict core business needs. This field has extremely high requirements for data privacy, result accuracy, and scenario adaptability. Traditional speculative decoding relies on independent draft models, which require extensive training with specialized medical corpora (such as electronic medical records, medical guidelines, and clinical trial data). The scarcity of medical data, privacy restrictions, and the difficulty of annotation further escalate the cost of customized training, far exceeding that of general-purpose scenarios. Furthermore, it is highly prone to discrepancies between candidate tokens and professional medical knowledge, potentially misleading critical decisions such as diagnostic assistance and medication recommendations. The introduction and fine-tuning of additional parameters required by self-specification methods not only increases compliance risks due to the sensitivity of medical data but may also disrupt the consistency of the target model's original medical knowledge, leading to unstable output results that fail to meet the stringent reliability requirements of clinical assistance and health management scenarios. Meanwhile, fixed computational paths cannot adapt to the dynamic differences in medical scenarios. For example, when faced with multi-dimensional information analysis of complex and difficult cases or simple Q&A in routine health consultations, the computational load cannot be dynamically adjusted. This may delay the processing time of emergency cases due to redundant computation, or it may waste medical resources. It is difficult to balance reasoning efficiency and result accuracy.

[0004] In the fintech field, the existing technical limitations of speculative decoding (SD) clash sharply with the compliance, real-time, and interpretability requirements of financial operations, giving rise to a series of key technical issues. The high training costs of traditional speculative decoding make it difficult for financial institutions to customize draft models for frequently changing financial data (such as market data, policy documents, and transaction records). Furthermore, the model iteration speed cannot keep up with the dynamic changes in financial scenarios, potentially causing candidate tokens to become disconnected from real-time financial information, affecting the accuracy of risk assessment, market prediction, and other decisions. The additional parameters and training processes introduced by self-speculation methods can undermine the interpretability of financial models. Since financial operations are subject to strict regulation, requiring traceable and auditable decision-making processes, the introduction of additional parameters may lead to ambiguous decision-making logic, failing to meet compliance requirements. It also makes it difficult to quickly adapt to different financial scenarios (such as credit approval, robo-advisors, and fraud detection) as a plug-and-play solution. Furthermore, fixed computing paths cannot cope with the complexity and dynamism of financial scenarios. For example, real-time risk warnings during periods of sharp market fluctuations require extremely fast reasoning speeds, while the generation of personalized financial plans requires detailed multi-dimensional calculations. The redundant calculations caused by fixed paths increase reasoning delays and affect the timeliness of key decisions. At the same time, they cannot dynamically allocate computing resources according to task priorities, which reduces the responsiveness of financial services and user experience, thus limiting their large-scale application in fintech scenarios. Summary of the Invention

[0005] The main objective of this invention is to provide a model speculation decoding method, apparatus, device, and storage medium based on dynamic layer skipping, which aims to solve the problems of traditional speculation decoding requiring high-cost training of external draft models, self-speculation methods having additional parameter training burden, and fixed model structures being difficult to adapt to dynamic scenarios, resulting in redundant calculations.

[0006] To achieve the above objectives, the present invention provides a model inference decoding method based on dynamic layer skipping, comprising: A target large language model is pre-constructed, and a draft model is defined using the target large language model; Obtain several skipping schemes, and construct a skipping pattern set based on the skipping schemes; Based on the current input state and the layer contribution of the target large language model, the optimal skipping pattern is selected from the skipping pattern set; The optimal skipping pattern is applied to the target large language model, and the draft model is updated according to the optimal skipping pattern. The updated draft model is used to perform G-step autoregressive inference to obtain several candidate tokens, and the number of acceptable tokens is determined based on these candidate tokens. The target large language model performs text decoding starting from a preset starting position based on the number of acceptable tokens.

[0007] Furthermore, to achieve the above objectives, the present invention provides a model inference decoding device based on dynamic layer skipping, comprising: The initial module for the draft model is used to pre-build the target large language model and define the draft model through the target large language model; The skip-layer mode module is used to obtain several skip-layer schemes and construct a skip-layer mode set based on the skip-layer schemes; The optimal skip module is used to select the optimal skip pattern from the skip pattern set based on the current input state and the layer contribution of the target large language model. The draft model update module is used to apply the optimal skipping pattern to the target large language model and update the draft model according to the optimal skipping pattern. The candidate token module is used to perform G-step autoregressive inference through the updated draft model to obtain several candidate tokens, and to determine the number of acceptable tokens based on the several candidate tokens; The model decoding module is used to perform text decoding of the target large language model starting from a preset starting position according to the number of acceptable tokens.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a model speculation decoding program based on dynamic skipping layers stored in the memory and executable on the processor, wherein when the model speculation decoding program based on dynamic skipping layers is executed by the processor, it implements the steps of the model speculation decoding method based on dynamic skipping layers as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a model speculation decoding program based on dynamic layer skipping, wherein when the model speculation decoding program based on dynamic layer skipping is executed by a processor, it implements the steps of the model speculation decoding method based on dynamic layer skipping as described above.

[0010] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business system platforms such as healthcare and fintech. It discloses a model speculation decoding method based on dynamic layer skipping, comprising: pre-constructing a target large language model and defining a draft model using the target large language model; obtaining several layer skipping schemes and constructing a layer skipping pattern set based on the schemes; selecting the optimal layer skipping pattern based on the current input state and the layer contribution of the target large language model; applying the optimal layer skipping pattern to the target large language model and updating the draft model based on the optimal layer skipping pattern; performing G-step autoregressive inference through the updated draft model to obtain several candidate tokens and determining the acceptable number of tokens based on these candidate tokens; and decoding the text from a preset starting position based on the acceptable number of tokens. This invention constructs a training-free draft model through dynamic layer skipping, selects the optimal layer skipping based on layer contribution evaluation, and iteratively generates verification candidate tokens, achieving efficient and lossless decoding. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a model inference and decoding method based on dynamic layer skipping in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the model inference and decoding method based on dynamic layer skipping of the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the model inference and decoding device based on dynamic layer skipping of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The model inference decoding method based on dynamic layer skipping provided in this invention can be applied to, for example... Figure 1In this application environment, the user terminal communicates with the server via a network. The server can pre-build a target large language model through the user terminal, define a draft model based on the target large language model; obtain several skipping schemes, construct a skipping pattern set based on the skipping schemes; select the optimal skipping pattern based on the current input state and the layer contribution of the target large language model; apply the optimal skipping pattern to the target large language model, update the draft model based on the optimal skipping pattern; perform G-step autoregressive inference through the updated draft model, obtain several candidate tokens, determine the acceptable number of tokens based on the several candidate tokens; starting from a preset starting position, the target large language model decodes the text based on the acceptable number of tokens. This invention constructs a training-free draft model through dynamic skipping, selects the optimal skipping layer through layer contribution evaluation, and iteratively generates verification candidate tokens to achieve efficient lossless decoding. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the model inference decoding method based on dynamic layer skipping provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the model inference decoding method based on dynamic layer skipping proposed in this invention includes the following steps: S100. Pre-construct a target large language model, and define a draft model through the target large language model; S200: Obtain several skipping schemes and construct a skipping mode set based on the skipping schemes; S300. Based on the current input state and the layer contribution of the target large language model, select the optimal skipping pattern from the skipping pattern set; S400. Apply the optimal skipping pattern to the target large language model, and update the draft model according to the optimal skipping pattern. S500: Perform G-step autoregressive inference using the updated draft model to obtain several candidate tokens, and determine the acceptable number of tokens based on the several candidate tokens; S600, The target large language model performs text decoding starting from a preset starting position according to the number of acceptable tokens.

[0016] In this embodiment, the first step is to construct the target large language model and define the draft model. Target large language models (LMMs) are... b The draft model (M) is the foundation of the entire acceleration solution and requires pre-training or fine-tuning to complete the process, possessing a complete multi-layer network structure and text generation capabilities. s The definition of SD breaks through the traditional approach of speculative decoding. It is not a small model trained separately, but is directly built on the target large language model. Its core logic is to select a non-continuous subset layer of the target large language model as the core structure of the draft model, which is implemented through subsequent dynamic layer skipping operations. The two completely share parameters, without the need to add additional parameters or carry out training procedures.

[0017] Next is the acquisition of skip-layer schemes and the construction of a skip-layer pattern set. A skip-layer scheme refers to the specific rules for determining which intermediate layers to skip in the target large language model, such as explicit layer selection strategies like "skip all even-numbered layers" or "skip layers 3, 6, and 9." By designing several such differentiated skip-layer schemes, they are integrated to form a skip-layer pattern set (P), where each element (π) in the set... k Each corresponds to an independent jump logic, providing a basis for subsequent dynamic selection of the optimal path.

[0018] Next comes the selection of the optimal skip-layer pattern. The core of this step is "dynamic adaptation," which requires combining two key pieces of information: first, the current input state, including the complexity of the input text, contextual redundancy, and the specific characteristics of the target task; second, the layer contribution of the target large language model. This is determined by evaluating the activation sparsity of each layer's attention matrix online to assess the actual effect of that layer on the current inference result. Activation sparsity reflects the redundancy of a layer; higher redundancy results in lower contribution. Based on these two pieces of information, the optimal skip-layer pattern (π) best suited for the current scenario is selected from the skip-layer pattern set. * The selection criteria are to maximize the efficiency of draft generation while ensuring a high acceptance rate for subsequent candidate tokens.

[0019] Next comes the draft model update and candidate token acquisition. Once the optimal skipping pattern is determined, it is applied to the target large language model. By dynamically skipping the intermediate layers specified in the pattern, an updated draft model (i.e., the M after skipping layers) is formed. b Because the draft model only retains a portion of the layers from the target large language model, and has significantly fewer layers than the original model, it exhibits extremely low latency when performing G-step autoregressive inference, and can quickly generate γ candidate tokens (x). t+1:γ). After generating candidate tokens, they need to be verified in parallel using the complete target large language model, according to the probability comparison rule of the standard inference decoding algorithm (P(x...). t (+i) Comparison with the original model probability) to determine the final acceptable number of tokens (α).

[0020] Among them, G-step autoregressive inference is the core operation of the draft generation stage. Its essence is to use the lightweight draft model built after skipping layers to continuously perform G rounds (G is a preset step number parameter, such as 3, 5, etc., which needs to be adjusted according to the requirements of inference efficiency and accuracy) of autoregressive token generation, and finally produce γ candidate token sequences (the number of γ corresponds to G-step inference).

[0021] Furthermore, "autoregressive inference" is a classic mode for large language models to generate text. It means that when the model generates each new token, it must rely on the previous one or several previously generated tokens as context input, and it cannot generate multiple tokens in parallel. This characteristic determines that there is a certain delay in the traditional generation process. "G-step" is a step limit for this autoregressive process. In the draft generation scenario, the model is no longer allowed to generate tokens one by one indefinitely. Instead, it stops after completing G rounds of continuous generation. This ensures that the candidate tokens have a certain length while minimizing the time cost of draft generation.

[0022] The specific process is as follows: The lightweight draft model first takes the currently determined text sequence (i.e., the token sequence up to time t during the inference process) as input to generate the (t+1)th candidate token; then, it takes "the original sequence + the (t+1)th token" as the new context to generate the (t+2)th candidate token; this process is repeated until G rounds of generation are completed, resulting in a complete candidate token sequence from t+1 to t+γ (γ=G). Since the draft model is built by skipping some intermediate layers of the target large model, its computational cost is much less than that of the full model. Therefore, even if the autoregressive approach is still used, the overall latency of G-step inference is significantly lower than that of the full model's autoregressive generation, thus saving time for the subsequent parallel verification stage and ultimately achieving the effect of accelerating the inference of large language models.

[0023] Finally, the text decoding of the target large language model is performed. Using the (t+α)th token as a preset starting position, the target large language model continues the subsequent text decoding process based on the determined number of acceptable tokens. The entire process strictly follows the speculative decoding paradigm, ensuring that the final sampled token distribution is completely consistent with the sampling results of the original target large language model, achieving lossless inference quality.

[0024] For example, in the healthcare business domain, the target large language model can be pre-built based on massive amounts of medical literature, clinical cases, and treatment guidelines, covering core scenario requirements such as disease diagnosis assistance, medical knowledge Q&A, and medical record generation. The draft model directly relies on this target model, selecting non-continuous subset layers related to medical text processing, professional terminology understanding, and clinical logic reasoning from the model's definitions, and can quickly respond to business requests without additional training. The skipping mode set can be pre-designed with various targeted solutions. For example, for simple medical common sense queries, a solution can be set to "skip 70% of non-core semantic layers"; for complex medical record analysis, a solution can be set to "skip only 20% of redundant syntax layers," while also incorporating customized skipping mode solutions for sub-scenarios such as specialist disease diagnosis and treatment and drug interaction queries. In practical applications, the system dynamically selects the optimal skipping mode based on the current input state: when the input is a consultation about symptoms of a common patient, by evaluating the activation sparsity of the attention matrix of each layer, it is found that the semantic understanding layer contributes the most, while the syntax analysis layer has high redundancy. Therefore, a high-proportion skipping scheme is selected to quickly generate candidate tokens. When the input is multi-dimensional information about complex cases, the core diagnosis and treatment logic layer and the feature extraction layer contribute significantly. Only a small number of redundant layers are skipped to ensure the professionalism and accuracy of the candidate tokens. The updated draft model generates candidate tokens that comply with medical standards (such as diagnostic suggestions, medication guidance, medical record keywords, etc.) through G-step autoregressive inference. These tokens are then validated in parallel by the complete target model. The number of acceptable tokens is determined based on the rigor standards of medical data. Finally, the target model completes text decoding from the starting position and outputs accurate and compliant medical service results. This not only improves the response speed of online consultations and medical record processing but also ensures the professionalism and security of medical inference.

[0025] In the fintech business field, the target large language model can be pre-built based on financial market data, trading rules, risk control systems, customer service scripts, and other data, focusing on core scenarios such as intelligent investment advisory, risk assessment, financial consulting, and trading assistance. The draft model is defined as a non-continuous subset layer in the target model related to financial semantic parsing, data trend analysis, and risk factor identification, adaptable to various financial businesses without additional parameter training. The skip-layer mode set includes multiple scenario-based solutions: for high-frequency, low-complexity needs such as real-time stock price queries and simple financial consulting, a solution is designed to "skip 60% of the non-core data processing layer"; for high-precision needs such as credit risk assessment and complex portfolio analysis, a solution is designed to "skip only 15% of the redundant statistical layer"; and it also covers dedicated skip-layer solutions for sub-scenarios such as credit card approval, insurance claims consulting, and blockchain transaction verification. In practical applications, the system selects the optimal solution based on the current input status and layer contribution: when the input is a customer's basic financial management needs consultation, the evaluation reveals that the financial terminology mapping layer and basic rule matching layer have high contributions, while the data depth mining layer has high redundancy. Therefore, a high-proportion skipping layer solution is selected to quickly generate candidate tokens. When the input is multi-dimensional data from a corporate loan application, the core risk assessment layer and financial indicator analysis layer have prominent contributions. Only a small number of irrelevant layers are skipped to ensure the accuracy of the candidate tokens. The updated draft model generates candidate tokens (such as investment advice, risk level assessment, and transaction operation prompts) through G-step autoregressive inference. After parallel verification by the complete target model, the acceptable number of tokens is determined. Finally, the target model decodes the text from a preset starting position, efficiently outputting results that meet financial regulatory requirements and accurately match business needs. This improves the efficiency of financial service response while ensuring transaction security and the rigor of risk control.

[0026] In one embodiment, step S100 includes: S101. Pre-construct the target large language model; S102. Define the non-continuous subset layer of the target large language model as a draft model; S103, wherein the number of layers in the draft model is less than the total number of layers in the target large language model.

[0027] In this embodiment, "pre-constructing the target large language model" is the foundation of the entire training-free self-predictive decoding process. Here, the target large language model refers to Large Language Models (LLMs, mathematically denoted as M). bThe construction process requires pre-training or fine-tuning the model to give it a complete multi-layer network architecture and standard text generation capabilities. Such models typically contain dozens to hundreds of Transformer structures (such as attention layers, Feed-Forward layers, etc.), which can perform token prediction and text decoding based on the input context and are the core carriers of all subsequent acceleration operations.

[0028] The key design of this invention, which "defines the non-continuous subset layer of the target large language model as a draft model," is its breakthrough over the bottleneck of traditional speculative decoding (SD) technology. Traditional SD requires training a small draft model separately, or adding extra parameters (such as the adapter / head) to the target model and fine-tuning it. In contrast, the draft model of this invention (mathematically represented as M)... s Completely based on the target model M b The construction requires no additional training or parameter additions: its core logic involves selecting a subset of non-contiguous layers from all layers in Mb to form the network structure of a draft model. For example, if the target model M... b If the total number of layers is L (e.g., 70 layers), then the draft model is M. s number of layers L s The number of layers will be strictly smaller than L (e.g., selecting 15 non-contiguous layers), and these layers directly reuse the original parameters of Mb, with no independent parameters or additional training required. This design maintains consistency with the target model while significantly reducing computational latency in the draft generation stage due to the reduced number of layers. It also achieves "zero training overhead" and "zero parameter overhead," providing a foundation for plug-and-play inference acceleration.

[0029] For example, in the healthcare business field, when pre-building a target large language model, the core needs of the healthcare industry are taken into account, and the model is trained and optimized based on massive amounts of compliant medical data. The training data includes authoritative medical literature (such as articles from journals such as The Lancet and The New England Journal of Medicine), standardized clinical case databases (containing different disease classifications, diagnosis and treatment procedures, and medication records), nationally issued diagnosis and treatment guidelines (such as disease diagnosis and treatment guidelines formulated by the National Health Commission), medical terminology dictionaries (such as ICD-10 International Classification of Diseases and UMLS Unified Medical Language System), and doctor-patient dialogue samples, etc., to ensure that the model has professional medical knowledge reserves and clinical logical reasoning capabilities, and can support core medical scenarios such as disease-assisted diagnosis, automatic medical record generation, medical knowledge Q&A, and medication risk warnings.

[0030] When defining the draft model, based on the characteristics of medical text processing, a non-contiguous subset of layers directly related to medical semantic understanding, key information extraction, and clinical rule matching is selected from all layers of the target large language model as the draft model. The number of layers in the draft model is strictly less than the total number of layers in the target large language model. For example, if the target model has 80 layers, the 25 layers responsible for medical terminology parsing, symptom-disease association mapping, and preliminary judgment of medication contraindications will be selected as the core layers of the draft model, while redundant syntax correction layers and general text style adjustment layers are excluded. This selection method ensures that the draft model can quickly process medical scenario inputs (such as patient symptom descriptions and doctor consultation needs) without additional training or parameter additions. It can directly generate candidate results based on the medical capabilities of the target model, meeting the low-latency requirements of scenarios such as online consultations and rapid preprocessing of emergency medical records, while also laying a precise foundation for subsequent full model validation.

[0031] In the fintech business, when pre-building the target large language model, we focus on the professionalism and compliance requirements of financial business scenarios, using multi-dimensional financial data as the training foundation. Training data includes historical financial market transaction data (price fluctuations and trading volume data of stocks, bonds, and funds), financial regulatory policy documents (such as compliance requirements issued by the central bank), corporate financial statement data, customer financial service interaction records (financial consultation, loan application dialogues), and risk control model parameters, ensuring that the target model possesses core capabilities such as financial market analysis, risk assessment, compliance verification, and intelligent customer service, making it adaptable to financial scenarios such as intelligent investment advisory, loan approval assistance, insurance claims calculation, and blockchain transaction traceability.

[0032] When defining the draft model, considering the core needs of financial data processing, a non-contiguous subset of layers related to financial semantic parsing, key indicator extraction, and basic risk identification is selected from the target large language model. The number of layers in the draft model is less than the total number of layers in the target model. For example, if the target model has 100 layers, the 30 layers responsible for financial term mapping (such as parsing "non-performing loan rate" and "annualized rate of return"), rapid customer demand classification (such as financial advice and loan applications), and preliminary screening of basic risk factors (such as keyword identification of customer credit records) will be determined as the core layers of the draft model. Redundant layers such as deep financial market prediction layers and complex derivative pricing layers are not included in the draft model. This design allows the draft model to quickly respond to high-frequency financial scenario needs (such as real-time stock price inquiries and basic financial advice generation). While ensuring processing speed, it leverages the financial expertise of the target model to generate preliminary candidate results, providing efficient upfront support for accurate risk verification and compliance review of the subsequent complete model, balancing the timeliness and professionalism of financial services.

[0033] In one embodiment, step S200 includes: S201. Design several sets of differentiated skipping schemes, each set of skipping schemes corresponding to the skipping rules of each layer in the target large language model; S202. Integrate the several groups of skipping schemes to construct a skipping pattern set; wherein each element in the skipping pattern set corresponds to an independent skipping scheme.

[0034] In this embodiment, designing differentiated skipping schemes and constructing a skipping pattern set are key preliminary steps for realizing the dynamic skipping mechanism. Its core objective is to provide rich candidate strategies for selecting the optimal computation path during subsequent runtime.

[0035] The first step is to "design several differentiated skip-layer schemes." Here, "differentiation" emphasizes that each scheme must possess different layer selection logic to adapt to different input scenarios (such as simple text generation, complex logical reasoning, etc.) and target large language models (i.e., M). b The difference in layer contribution. Each skip-layer scheme explicitly corresponds to M. b The skipping rules for each layer strictly follow the established logic and mainly fall into three categories: First, "skip even-numbered layers," which means skipping layers according to their sequence number (e.g., layer 1, layer 2... layer L, where L is M). b (Total number of layers), skip all layers with even numbers, such as M. b When there are 10 layers in total, skip layers 2, 4, 6, 8, and 10; the second option is to "skip consecutive layers with preset sequence numbers," which means specifying a consecutive range of layer numbers in advance and skipping them. For example, if you set it to skip layers 5-8, then M... b Layers 5, 6, 7, and 8 are not included in the draft model calculation; thirdly, "skip discrete layers with preset numbers," which means pre-specifying several non-contiguous layer numbers and skipping them, for example, setting to skip layers 3, 6, and 9, these layers in M b The layers are distributed in a dispersed manner within the sequence. By designing multiple sets of such differentiated schemes, the computational pruning requirements under different layer redundancy scenarios can be covered.

[0036] The subsequent "integration of several skip-level schemes into a skip-level pattern set" involves systematically aggregating all the independent skip-level schemes described above into a set containing multiple skip-level logics, namely, the skip-level pattern set P. Each element in the set (represented mathematically as π) k Each π(k) represents a unique set of skipping schemes. For example, π1 corresponds to the "skip even-numbered layers" scheme, π2 corresponds to the "skip layers 3, 6, and 9" scheme, and π3 corresponds to the "skip layers 5-8" scheme, etc. This set-based design allows for rapid selection of the optimal skipping pattern (πk) that best suits the current input state after evaluating layer contributions during runtime. *This provides a flexible and efficient basis for dynamically adjusting the calculation path, while avoiding the problem of insufficient adaptability caused by hard-coded single jump rule.

[0037] For example, in the healthcare business field, when designing differentiated skipping schemes, specific skipping rules are formulated by closely considering the text complexity, information accuracy requirements, and response time requirements of medical scenarios. For simple medical scenarios (such as consultation on common cold symptoms or inquiries about common medication usage), a basic scheme of "skipping even-numbered layers" is designed: if the target large language model has 60 layers, layers 2, 4, 6...60 are skipped, retaining only odd-numbered layers for rapid processing of general medical semantics, meeting the low-latency response requirements of high-frequency, low-complexity scenarios. For medium-complexity scenarios (such as daily management advice for chronic diseases or interpretation of basic physical examination reports), a scheme of "skipping layers with preset sequential numbers" is designed, for example, skipping layers 15-20 (general text style optimization layers) and layers 35-40 (non-medical layers). The system includes a domain knowledge association layer, retaining core layers directly related to chronic disease diagnosis and treatment logic and physical examination indicator interpretation, balancing accuracy with efficiency. For highly complex scenarios (such as preliminary analysis of difficult cases and integration of multidisciplinary consultation opinions), a "skip preset sequence number discrete layer" scheme is designed. For example, only layer 8 (general syntax error correction layer), layer 25 (non-medical data association layer), and layer 48 (low-priority medical literature citation layer) are skipped, maximizing the retention of key layers related to disease diagnosis reasoning and interdisciplinary medical knowledge integration, ensuring the accuracy of complex medical information processing. The above schemes, along with more subdivided scenario schemes (such as pediatric disease-specific skipping schemes and tumor treatment consultation skipping schemes), are integrated to construct a set of medical-specific skipping patterns. Each element of the set corresponds to an independent scheme, which can be dynamically invoked according to actual medical business scenarios (such as online consultation, medical record preprocessing, and medical knowledge Q&A), ensuring both the professionalism of medical reasoning and adapting to the efficiency needs of different scenarios.

[0038] In the fintech business, when designing differentiated layer skipping solutions, we formulate appropriate layer skipping rules based on the business type of the financial scenario (such as high-frequency queries, accurate assessments, and compliance verification), data complexity, and risk control requirements. For high-frequency, low-complexity scenarios (such as real-time stock price queries and basic information consultations on wealth management products), we design a "skip even-numbered layers" solution: if the target large language model has 80 layers, we skip layers 2, 4, 6...80, retaining odd-numbered layers for fast parsing of basic financial semantics to meet the high-frequency query requirements of thousands of queries per second. For medium-complexity scenarios (such as preliminary personal loan review and wealth management plan recommendations for ordinary customers), we design a "skip pre-ordered consecutive layers" solution, for example, skipping layers 20-25 (complex financial derivatives pricing layer) and layers 50-55 (cross-market deep correlation analysis layer), retaining personal... The core layers related to credit reporting interpretation and basic wealth management rule matching improve the efficiency of review and recommendation while controlling risks. For high-complexity and high-risk scenarios (such as corporate credit risk assessment and large-scale investment portfolio strategy formulation), a "skip preset sequence discrete layer" scheme is designed. For example, only the 12th layer (redundant explanation layer of general financial terms), the 38th layer (low-priority market dynamic analysis layer), and the 65th layer (non-compliant data association layer) are skipped, while key layers related to corporate financial indicator analysis, investment risk factor verification, and financial regulatory compliance matching are retained to ensure the accuracy and compliance of high-risk business processing. These schemes and subdivided scenario schemes (such as insurance claims consultation skipping schemes and blockchain transaction traceability skipping schemes) are integrated to form a set of skipping models in the fintech field. Each element corresponds to an independent scheme, which can be flexibly called according to actual business scenarios (such as intelligent customer service, risk review, and investment advisory services), improving the efficiency of financial services while strictly adhering to risk control and compliance bottom lines.

[0039] In one embodiment, step S300 includes: S301. During the decoding process, obtain the activation sparsity of the attention matrix of each layer of the target large language model; S302. Measure the contribution of the corresponding layer in the current context according to the activated sparse metric; S303. Based on the contribution level and the current input state, select the optimal skipping mode from the skipping mode set.

[0040] In this embodiment, "obtaining the activation sparsity of the attention matrix of each layer of the target large language model" is the core data source for dynamically evaluating the contribution of each layer. This operation is completed entirely by the inference process of the target large language model itself, without the need to introduce additional external computing modules. The attention matrix is ​​a core component of the Transformer structure in LLMs. Its function is to calculate the association weights between different tokens in the input text. The "activation sparsity" is used to describe the sparsity of the effective activation elements (i.e., the weight values ​​that actually contribute to the current context association) in the matrix. The higher the activation sparsity, the more elements in the attention matrix of that layer are in a low-contribution or no-contribution state, indirectly reflecting the strong redundancy of that layer in the current inference scenario.

[0041] The key innovation of this invention, which quantifies the contribution of activation sparsity layers, is based on the aforementioned logic. Specifically, contribution is negatively correlated with activation sparsity: when the activation sparsity of a certain layer's attention matrix is ​​high, it indicates that the layer has a lot of redundancy in processing the current context information, providing less effective information for the inference result, and its contribution is quantified as low. Conversely, if the activation sparsity is low, it indicates that the proportion of effective activation elements in the layer's attention matrix is ​​high, providing key related information for the current inference, and its contribution is quantified as high. This quantification process is performed entirely online, adapting in real time to the context features of the current input text, avoiding the limitations of traditional fixed-layer skipping rules that are divorced from real-world scenarios.

[0042] Finally, "selecting the optimal skipping pattern based on contribution and current input state" is based on the above quantitative results, combined with the specific characteristics of the current input text (such as input complexity, context redundancy, target task type, etc.), to select the most suitable skipping scheme (i.e., the optimal skipping pattern π) from the preset skipping pattern set (P). * The core criterion for selection is to maximize draft generation efficiency while ensuring a high acceptance rate for subsequent candidate tokens. For example, when quantization results show that many layers have low contribution and the current input is a simple text generation task, a skip-layer mode that skips more low-contribution layers will be selected to further reduce draft generation latency. If some layers have high contribution and the current input is a complex logical reasoning task, a mode that skips only a few redundant layers will be selected to ensure that the draft model retains key reasoning information and maintains a high acceptance rate. Through this dynamic selection process, precise allocation of computing resources is achieved, ensuring both inference quality and maximizing acceleration.

[0043] For example, during the decoding process of healthcare business, when the target large language model processes medical text (such as patient symptom descriptions, medical record fragments, and medical consultation questions), it acquires the activation sparsity of the attention matrix of each layer in real time. For instance, when processing input such as "a patient has a recurring cough with a low-grade fever for 3 days and a history of asthma," the layer in the model responsible for associating medical symptoms and extracting medical history information will focus its attention matrix on key medical entities such as "cough," "low-grade fever," and "history of asthma," resulting in low activation sparsity (i.e., attention is concentrated on core information); while the layer responsible for general text syntax optimization and associating non-medical domain knowledge has higher activation sparsity (i.e., limited contribution to the current medical context).

[0044] Subsequently, the system quantifies the contribution of each layer based on activation sparsity: the lower the activation sparsity, the more focused the layer's attention is on the key information of the current medical context, and the higher its contribution; conversely, the higher the activation sparsity, the lower the contribution. Taking the above input as an example, the medical symptom analysis layer and the medical history association layer are judged as high-contribution layers due to their low activation sparsity, while the general syntax layer and the non-medical knowledge layer are judged as low-contribution layers due to their high activation sparsity.

[0045] Finally, the optimal skipping mode is selected by combining the contribution and the current input state: If the current input is a simple consultation about a common disease (such as "how to relieve a cold"), the high contribution layers are concentrated in the basic medical knowledge layer, and the low contribution layers account for a large proportion. In this case, the solution that "skips most of the low contribution general layers" (such as skipping the general syntax and non-medical knowledge layers in even-numbered layers) will be selected from the skipping mode set. If the current input is a complex and difficult case description (such as "symptoms of multi-organ involvement caused by rare gene mutations"), the high contribution layers cover the rare disease knowledge base layer, the multi-organ association analysis layer, etc., and the low contribution layers are very few. In this case, the solution that "skips only a very few redundant layers" (such as skipping only the discrete low contribution general text processing layer) will be selected to ensure the accuracy of complex medical information processing and avoid redundant calculations.

[0046] In the decoding process of fintech business, when the target large language model processes financial text (such as customer financial advice, corporate loan application data, and market transaction instructions), it captures the activation sparsity of the attention matrix of each layer in real time. For example, when processing input such as "a small and micro enterprise applies for a 5 million yuan business loan, with an average annual revenue growth of 15% in the past 3 years and a debt-to-asset ratio of 40%", the layer in the model responsible for analyzing corporate financial indicators and extracting credit risk factors will focus its attention matrix on core financial data such as "5 million yuan", "15% revenue growth", and "40% debt-to-asset ratio", resulting in low activation sparsity. On the other hand, the layer responsible for redundant interpretation of general financial terms and analysis of non-credit-related market dynamics will have high activation sparsity.

[0047] Next, the system quantifies the contribution of each layer based on the activation sparsity: layers with low activation sparsity (such as the financial indicator analysis layer and the credit risk assessment layer) are judged as high contribution layers because they are deeply involved in the processing of key information in the current financial context; layers with high activation sparsity (such as the general terminology explanation layer and the non-credit market analysis layer) are judged as low contribution layers because they have limited support for the current credit application scenario.

[0048] Finally, the optimal skipping mode is selected by combining the contribution level and the current input status: If the current input is a simple customer financial management query (such as "how to calculate the expected return of a money market fund"), the high contribution level is concentrated in the basic financial management rules layer, and there are many low contribution levels. The solution that "skips a large number of low contribution complex financial analysis layers" (such as skipping continuous derivative pricing and cross-market correlation layers) will be selected from the skipping mode set. If the current input is a high-risk large-scale investment portfolio adjustment instruction (such as "allocate 100 million yuan to an overseas emerging market fund"), the high contribution level includes the market risk assessment layer and the cross-border compliance verification layer, and there are very few low contribution levels. The solution that "only skips discrete low contribution general layers" will be selected, which improves decoding efficiency while ensuring the accuracy and compliance of investment decisions.

[0049] In one embodiment, step S400 includes: S401. Apply the optimal skipping pattern to the target large language model; S402. Skip redundant layers of the target large language model according to the optimal skipping mode; S403. Define the non-skipped layer after performing a skip layer in the target large language model as a draft model.

[0050] In this embodiment, "applying the optimal skipping pattern to the target large language model" is a key step connecting dynamic layer evaluation and draft model construction. The optimal skipping pattern is the skipping scheme (π) adapted to the current scenario, which was previously selected from the skipping pattern set (P). * The application process is not for M. b No modifications are made to the model structure or parameters, but rather at the inference runtime level, according to π. * The skip rules defined in M b The layer calculation process is dynamically adjusted; for example, if π * If the option is "skip layers 3, 6, and 9", then in subsequent calculations, attention calculations, feed-forward calculations, and other operations in these three layers will be skipped, and only the normal operation of other layers will be triggered. This ensures that the layer skipping operation is efficient and reversible, and does not affect M. bThe integrity of the task itself. Furthermore, the optimal skip-layer pattern must satisfy the following: retain all layers with a contribution ≥ 0.6, and skipped layers ≤ 50% of the total number of layers (balancing efficiency and accuracy); if the input is a highly complex task (such as difficult medical cases or large-scale financial credit assessments), adjust the contribution threshold to 0.5 and reduce the number of skipped layers to below 30% of the total number of layers; when multiple patterns meet the criteria in the skip-layer pattern set, select the pattern with the lowest average contribution of skipped layers to ensure the most thorough reduction of redundant computation, filling logical gaps.

[0051] The core purpose of applying the skipping layer pattern is to "skip redundant layers of the target large language model according to the optimal skipping layer pattern." Here, "redundant layers" specifically refers to layers previously evaluated by the sparsity of the attention matrix activation and those that contribute little to the current contextual reasoning. Because the optimal skipping layer pattern π * It is determined based on layer contribution and the current input state, and the layers to be skipped are defined by M. b Redundant layers in the model are highly matched; for example, when certain layers are judged to be redundant due to high activation sparsity, π * These layers will be included in the skipping scope, and their computation process will be directly skipped to achieve precise pruning of redundant calculations. This operation avoids the waste of computational power caused by redundant layers in traditional fixed-structure models, and does not rely on additional trained draft models like traditional speculative decoding (SD), relying entirely on M. b It performs computational optimization on its own.

[0052] Finally, defining the non-skipped layers after skipping layers in the target large language model as the draft model clarifies the draft model (M). s The construction logic of M in this invention differs from the independently trained small draft models in traditional SD. s Essentially M b The "compute subset", after performing the skip layer, M b The layers that are not skipped (i.e., non-skipped layers) form a complete, independently runnable inference path, which is the draft model M. s For example, if M b There are 20 layers in total, and π is executed. * After skipping 5 redundant layers, the remaining 15 non-skipped layers constitute M. s The key is M. s With M b Fully shared parameters, requiring no additional parameters or fine-tuning, and M s number of layers (L) s It must be less than M. b The total number of layers (L) makes M sIn subsequent generation of candidate tokens, latency can be significantly reduced due to decreased computational load, while also benefiting from the reuse of M... b The parameters ensure consistency with the target model, laying the foundation for a high acceptance rate in subsequent parallel validation.

[0053] For example, in the healthcare business domain, when applying the optimal skipping pattern to the target large language model, it closely combines the core needs and data characteristics of the current medical scenario to ensure that the skipping operation improves efficiency without affecting the professionalism of medical reasoning. For instance, when processing low-complexity inputs such as "consultation on medication for colds and fever" from ordinary patients, the previously selected optimal skipping pattern is "skip 70% of non-core general layers." When applying this pattern, the system will accurately locate redundant layers in the target model that are weakly related to medical semantics, such as the general text sentiment analysis layer, the non-medical domain knowledge association layer, and the complex grammar error correction layer, and directly skip these layers that do not substantially support consultation on cold medication. For high-complexity inputs such as "integration of multidisciplinary consultation opinions for difficult tumor cases," the optimal skipping pattern is "skip only 10% of discrete redundant layers." In this case, only a few redundant layers such as the low-priority basic medical knowledge layer and the non-tumor domain symptom analysis layer in the target model will be skipped, while retaining the core layers such as the tumor subtyping analysis layer, the interdisciplinary treatment plan matching layer, and the drug interaction assessment layer to the greatest extent.

[0054] After skipping redundant layers, the remaining non-skipped layers in the target large language model are defined as draft models. Taking an 80-layer medical target model as an example, when processing cold medication consultations, the remaining 24 layers after skipping redundant layers (such as the basic drug knowledge layer, common disease medication rules layer, and medication dosage safety verification layer) constitute the draft model; when processing tumor consultation opinion integration, the remaining 72 layers after skipping redundant layers (such as the tumor pathology analysis layer, multidisciplinary diagnosis and treatment logic layer, and targeted drug efficacy evaluation layer) constitute the draft model. These draft models rely entirely on the medical professional parameters of the target model, requiring no additional training. They can quickly respond to the input needs of different medical scenarios, such as rapidly generating medication suggestion candidate tokens in cold consultation scenarios and accurately generating diagnosis and treatment opinion candidate content in tumor consultation scenarios. They also lay a high-quality foundation for the parallel validation of the subsequent complete model, taking into account both the timeliness and accuracy of medical services.

[0055] In the fintech business, when applying the optimal skipping pattern to the target large language model, the risk level and data complexity of the financial scenario are taken into account to achieve precise screening and skipping of redundant layers. For example, when processing high-frequency, low-risk inputs such as "real-time query of the current price of a stock," the previously determined optimal skipping pattern is "skip 60% of non-core data processing layers." When applying this pattern, the system will skip redundant layers in the target model that are unrelated to real-time stock price queries, such as complex financial derivatives pricing layers, corporate in-depth financial analysis layers, and cross-market risk correlation layers, retaining only the layers related to securities code parsing, real-time market data mapping, and basic market information display. However, when processing high-complexity, high-risk inputs such as "risk assessment of a company applying for a 100 million yuan loan," the optimal skipping pattern is "skip only 15% of discrete redundant layers." In this case, only a few redundant layers in the target model, such as low-priority general financial terminology interpretation layers and non-credit-related market dynamic analysis layers, will be skipped, while core layers such as corporate financial indicator parsing layers, credit risk factor verification layers, and financial regulatory compliance matching layers will be retained.

[0056] After the redundant layer skipping operation is completed, the non-redundant layers in the target large language model that were not skipped are defined as the draft model. Taking a 100-layer financial target model as an example, when processing stock price queries, the remaining 40 layers after skipping redundant layers (such as the security identifier resolution layer, real-time data interface adaptation layer, and basic market data display logic layer) constitute the draft model, which can quickly generate candidate tokens related to stock price information; when processing corporate credit risk assessment, the remaining 85 layers after skipping redundant layers (such as the corporate asset and liability analysis layer, repayment ability prediction layer, and credit default risk warning layer) constitute the draft model, which can accurately generate candidate content related to risk assessment. This draft model, built based on the optimal skipping layer pattern, not only relies on the financial expertise of the target model to ensure the accuracy of candidate results, but also significantly improves decoding speed by skipping redundant layers. It can meet the low-latency requirements of high-frequency queries and the accuracy requirements of high-risk businesses in financial scenarios, while providing efficient support for the parallel verification of the subsequent complete model.

[0057] In one embodiment, step S500 includes: S501. Perform G-step autoregressive inference using the updated draft model to generate several candidate tokens; S502. Input the plurality of candidate tokens into the target large language model, and perform parallel verification of the plurality of candidate tokens through the target large language model to obtain the verification result of each candidate token; S503. Based on the verification results, calculate the generation probability of the target large language model for each candidate token and the generation probability of the draft model for each candidate token. S504. Based on the standard speculative decoding algorithm, calculate the ratio of the generation probability of the target large language model for each candidate token to the generation probability of the draft model for each candidate token. S505. Determine the validity of each candidate token based on the ratio, and count the number of acceptable tokens based on the validity.

[0058] In this embodiment, "performing G-step autoregressive inference using the updated draft model to generate several candidate tokens" is the initial step in the self-predictive decoding process. Here, the draft model (M...) s ) is the set of non-skipped layers obtained after processing the target large language model (Mb) using the optimal skipping pattern, and its number of layers is less than M. b The computational latency is significantly reduced. Autoregressive inference is a classic method for LLMs to generate text, where the model predicts the next token based on the token generated in the previous time step, and "G steps" refers to performing this prediction operation G times consecutively, ultimately generating γ candidate tokens (denoted as xt+1:γ). Since M... s reuse M b With fewer parameters and fewer layers, this step can quickly generate candidates while ensuring consistency between the candidate tokens and the target model's generation logic, laying the foundation for accelerated verification in the future.

[0059] "Inputting several candidate tokens into the target large language model and performing parallel verification" utilizes the complete target model M b The computing power advantage of M s The core step in verifying the accuracy of the generated candidate tokens is to perform parallel verification, unlike the traditional serial method of verifying token by token in autoregressive inference. This invention uses parallel verification, that is, inputting γ candidate tokens into M at once. b Through M b The complete layer structure (without skipping layers) simultaneously calculates the rationality of each candidate token in the current context, and finally outputs the verification result of each candidate token (including key information such as the probability distribution of token generation and context matching degree). This parallel processing method significantly shortens the time spent in the verification stage, which is a key design to improve the overall inference efficiency, and also conforms to the core logic of speculative decoding (SD) to balance speed and accuracy through "fast generation + parallel verification".

[0060] The subsequent calculation of the two generation probabilities and their ratio serves as a quantitative basis for judging the validity of candidate tokens. First, based on the parallel verification results described above, two key probability values ​​are extracted: one is the generation probability (P0) of the target large language model Mb for each candidate token. Mb (x t+i The first is the complete model, which considers the token reasonable in the current context; the second is the draft model M. s The generation probability (P) of each candidate token Ms (x t+i The value represents the confidence level of the simplified model in generating the token. Then, following the standard speculative decoding algorithm, the ratio (P) is calculated. Mb (x t+i) / P Ms (x t+i This ratio reflects the difference in acceptance of the same token between the complete model and the simplified model. The closer the ratio is to 1, the stronger the acceptance of the complete model. s The generated token and M b The more consistent the generation logic, the higher the effectiveness.

[0061] Finally, "determining validity based on the ratio and counting the number of acceptable tokens" is the final step in screening candidate tokens. Specifically, a reasonable probability ratio threshold is set (conforming to the probability comparison rules of speculative decoding). If the probability ratio of a candidate token reaches or exceeds the threshold, the token is considered valid and acceptable; if it does not reach the threshold, it is considered invalid and discarded. For example, if the threshold is set to 0.8, a token with a ratio of 0.9 is accepted, and one with a ratio of 0.7 is rejected. The total number of all valid candidate tokens is the number of acceptable tokens (α). The core function of this step is to ensure that the tokens that ultimately enter the subsequent decoding process fully conform to the original target model M. b The generation distribution ensures the lossless quality of reasoning, while accepting valid tokens reduces redundant calculations and further improves overall reasoning efficiency.

[0062] For example, in the healthcare business domain, the updated draft model will perform G-step autoregressive inference based on the specific scenario to generate candidate tokens. For instance, in an online consultation scenario, for a patient's input of "persistent headache with blurred vision for 3 days", the draft model (built by skipping layers from the target medical model, retaining core layers such as headache symptom association and neurological symptom analysis) performs 3-step autoregressive inference to generate candidate tokens related to the diagnosis and treatment recommendations, such as "recommendation to undergo a head CT scan", "screen for risk of increased intracranial pressure", and "prioritize booking a neurology clinic".

[0063] Subsequently, these candidate tokens are input into the complete target large language model for parallel validation. The target model, relying on its complete medical knowledge system (covering neurological disease diagnosis and treatment guidelines, indications for imaging examinations, departmental triage standards, etc.), validates the professionalism, compliance, and suitability of each candidate token. For example, it determines whether "recommending a head CT scan" conforms to the routine examination path for headache accompanied by blurred vision, and whether "screening for increased intracranial pressure" is a key risk point corresponding to this symptom. Finally, it outputs the validation results for each candidate token (such as "complies with diagnosis and treatment guidelines", "needs to supplement the applicable population description", "low correlation with symptoms").

[0064] Based on the verification results, the system will calculate the generation probability of each candidate token for the target model and the draft model respectively: Assuming that the generation probability of "suggesting a head CT scan" is 0.85 for the target model (representing the degree of rationality of the suggestion in medical logic), the generation probability of the draft model for this token is 0.72 (representing the rationality of the draft model's judgment based on simplified calculation); for "prioritizing the neurology outpatient clinic", the generation probability of the target model is 0.91, and the generation probability of the draft model is 0.78.

[0065] Following the standard speculative decoding algorithm, the ratio of the probabilities of the two tokens is calculated: for example, the probability ratio of "recommend a head CT scan" is 0.85 / 0.72≈1.18, and the ratio of "prioritize booking a neurology appointment" is 0.91 / 0.78≈1.17. The system will determine the validity based on a threshold set for the medical scenario (e.g., a ratio ≥1.05 is considered valid, ensuring that candidate tokens meet medical rigor requirements). Both tokens mentioned above meet the threshold requirements. However, if the ratio of a candidate token "recommend taking common painkillers for relief" is 0.92 / 0.88≈1.05 (or lower than the threshold), its validity needs to be further evaluated in conjunction with the symptom risk level. Finally, through validity statistics, the number of acceptable tokens is determined (e.g., 2 acceptable tokens in the above scenario), providing a basis for the subsequent target model to accurately output treatment recommendations, ensuring the professionalism of medical recommendations while avoiding response delays caused by redundant calculations.

[0066] In the fintech business, the updated draft model will perform G-step autoregressive inference to generate candidate tokens based on the needs of financial scenarios. For example, in the personal credit review scenario, for the user's input of "monthly salary of 8,000 yuan, no mortgage, applying for a 200,000 yuan consumer loan", the draft model (built by skipping layers from the target financial model, retaining the core layers such as income-to-debt ratio calculation, credit limit assessment, and basic risk screening) will perform 4 steps of autoregressive inference to generate candidate tokens related to credit approval and terms setting, such as "suggested credit limit of 150,000 yuan", "maximum loan term of 3 years", "requires supplementary bank statements for the past 6 months", and "annualized interest rate reference of 4.8%".

[0067] These candidate tokens are input into the complete target large language model for parallel verification. The target model verifies each candidate token based on financial regulatory policies (such as regulations on the upper limit of consumer loan amounts), credit risk control models (rules for matching income with credit limits), market interest rate pricing standards, etc. For example, it determines whether "suggested credit limit of 150,000 yuan" conforms to the industry's conventional credit granting logic of "10-15 times monthly income", and whether "annualized interest rate of 4.8%" is within the current market compliant interest rate range, and outputs the verification results (such as "complies with credit granting rules", "interest rate pricing is reasonable", "supplementary materials required for compliance").

[0068] Based on the verification results, the generation probabilities of the target model and the draft model were calculated respectively: it is assumed that the generation probability of the target model for "suggested credit limit of RMB 150,000" is 0.92, and that of the draft model is 0.81; the generation probability of the target model for "requires supplementary bank statements for the past 6 months" is 0.95, and that of the draft model is 0.86.

[0069] The ratios are calculated using the standard inference decoding algorithm: the ratio for "suggested credit limit of 150,000 yuan" is 0.92 / 0.81≈1.14, and the ratio for "requires supplementary bank statements for the past 6 months" is 0.95 / 0.86≈1.10. Financial scenarios typically set higher rigor thresholds (e.g., ratios ≥ 1.08), and both tokens above meet the validity requirements. If the ratio for a candidate token "maximum loan term of 5 years" is 0.88 / 0.82≈1.07 (below the threshold), it is deemed invalid (because it does not comply with the implicit regulatory requirement of "maximum 3-4 years for consumer loans"). The final number of acceptable tokens is counted (e.g., 3 in the above scenario) to support the subsequent target model in outputting compliant and accurate credit approval results, improving approval efficiency while strictly adhering to the bottom line of financial risk control and regulatory compliance.

[0070] In one embodiment, step S600 includes: S601. Set the acceptable number of tokens to α; S602. Set the initial token position before executing the G-step autoregressive inference to t; S603. Update the token position of the current inference progress to the (t+α)th token, that is, the (t+α)th token is the preset starting position of text decoding; S604. Starting from the (t+α)th token position, repeatedly execute the process of filtering the optimal skipping mode based on the current input state, updating the draft model, generating candidate tokens for verification, and determining the number of acceptable tokens. S605. Repeat the above process until the target large language model completes text decoding.

[0071] In this embodiment, "setting the acceptable number of tokens as α" and "setting the initial token position as t" are key operations for clarifying the starting state of the loop. Here, α is the total number of valid candidate tokens selected after parallel verification by the target large language model in the previous round, representing the inference results that can be directly reused in this round; t is the token position before executing G-step autoregressive inference, that is, the termination position of the previous round of decoding process. For example, if the previous round decoded to the 10th token, then t=10. These two parameters together determine the starting point of the new round of decoding.

[0072] "Update the token position of the current inference progress to the (t+α)th token" essentially advances the decoding progress based on the valid results of the previous round, avoiding redundant calculations. Since α candidate tokens have already passed M... b The verification results are consistent with the generation distribution of the original model, so there is no need to re-infer these tokens. Therefore, the decoding starting point is directly updated from t to t+α. For example, when t=10 and α=3, the new round of decoding starts from the 13th token (t+α=13). This step not only ensures the lossless quality of inference, but also further improves the overall decoding efficiency by reusing effective results.

[0073] "Repeating the core process starting from the (t+α)th token position" is the core logic for maintaining continuous decoding. The "core process" here refers to the previously completed dynamic optimization chain: First, based on the new input state (new context) starting from the (t+α)th token, re-evaluate M. b The attention matrix activation sparsity of each layer is used to select the optimal skipping mode (π) that best suits the current scenario. * ); then based on π * Update draft model (M) s (i.e., skip M) b The new redundant layers in the M are composed of non-skipped layers. sThen, G-step autoregressive inference is performed through the new Ms to generate a new round of γ candidate tokens; finally, Mb is used to perform parallel verification of these candidate tokens, recalculate the probability ratios and determine the new number of acceptable tokens α', completing a new cycle.

[0074] The phrase "repeatedly execute the above process until text decoding is complete" clarifies the termination condition for the entire decoding process. Since text generation in large language models is token-by-token, multiple rounds of this "update starting point - dynamic optimization - generation verification" loop are needed to gradually advance the token position (from t→t+α→t+α+α'→……) until text meeting the termination condition is generated (such as reaching the preset text length, generating the end marker , etc.), ultimately completing the full text decoding. This loop mechanism not only continues the core advantages of "zero training overhead" and "dynamic adaptation," but also ensures that the entire decoding process maintains a highly efficient acceleration state while guaranteeing lossless inference quality through continuous progress updates and process reuse, perfectly adapting to the various scenarios requiring large language models, from short text generation to long text creation.

[0075] For example, in the healthcare business field, this cyclical decoding process revolves around the professionalism and continuity requirements of medical text processing, ensuring that results such as treatment recommendations and medical record generation are accurate and consistent with clinical logic. For instance, in the intelligent medical record generation scenario, the number of acceptable tokens is first set to α: assuming the draft model generates candidate tokens such as "Patient admitted for 'recurrent chest pain for 2 weeks, with a history of hypertension for 5 years'", after verification by the target model, the number of acceptable tokens conforming to the medical record specifications is α=8. Then, the initial token position before executing the G-step autoregressive inference is set to t=0 (i.e., the beginning of the medical record). At this point, the token position of the inference progress is updated to t+α=8, and the 8th token (such as the connecting position after "5-year history of hypertension") becomes the preset starting position for the next round of text decoding.

[0076] Starting from the 8th token position, the system will repeat the complete process: First, based on the current input state (i.e., the generated medical record content "Patient admitted for 'recurrent chest pain for 2 weeks, with a history of hypertension for 5 years"), the contribution of each layer of the target model will be evaluated, and the optimal skipping mode suitable for the "medical history supplementation and symptom association analysis" scenario will be selected (e.g., skipping only the general text decoration layer and retaining the medical history parsing and symptom-disease association layers); then, the draft model will be updated based on this mode, and G-step autoregressive inference will be performed through the new draft model to generate candidate tokens such as "poor blood pressure control, up to 160 / 100 mmHg, no history of diabetes"; then, after parallel verification by the target model, the number of acceptable tokens in the new round will be determined to be α=6, and the token position will be updated to 8+6=14.

[0077] This process is repeated in a loop. In each round, the skipping strategy and draft model are dynamically adjusted based on the currently generated medical text as context. Tokens that conform to medical record specifications are continuously generated and verified until the target model completes the decoding of the complete medical record (covering all elements such as chief complaint, present illness, past medical history, and physical examination recommendations). This ensures the efficiency of medical record generation and, through round-by-round verification and position iteration, ensures the clinical accuracy and logical coherence of the medical record content, avoiding information omissions or errors caused by one-time generation.

[0078] In the fintech business, this cyclical decoding process closely adheres to the compliance and risk control requirements of financial texts, ensuring that outputs such as robo-advisory recommendations and credit approval reports are accurate and compliant with regulatory standards. For example, in the scenario of generating corporate credit approval reports, the acceptable number of tokens α is first set: the draft model generates candidate tokens such as "A technology company applies for a 30 million yuan business loan, with revenue of 210 million yuan and net profit of 32 million yuan in 2023." After verification by the target model (checking the authenticity of revenue and net profit, and the compliance of the loan amount), the acceptable number of tokens α=10 is determined. The initial token position t=0 (the beginning of the report) is updated to t+α=10, and the 10th token (such as the analysis connection position after "net profit of 32 million yuan") becomes the preset starting position for the next round of decoding.

[0079] Starting from the 10th token position, the system repeats the process: based on the generated basic financial and loan demand information of the enterprise, it evaluates the contribution of each layer of the target model and selects the optimal skipping mode that is suitable for the "in-depth analysis of financial indicators and risk assessment" scenario (such as skipping the general financial terminology explanation layer while retaining the financial ratio calculation and debt repayment ability analysis layer); after updating the draft model, it performs G-step inference to generate candidate tokens such as "asset-liability ratio 45%, current ratio 1.8, in line with the industry average level"; after verification by the target model (verifying the accuracy of financial ratio calculation and its matching degree with industry standards), it determines a new round of α=7, and the token position is updated to 10+7=17.

[0080] Through this iterative process, each round dynamically optimizes the skip-layer strategy based on the generated financial text context, continuously generating and verifying compliant candidate tokens until a complete credit report is produced, including corporate financial analysis, risk level assessment, and approval recommendations. This process not only improves report generation efficiency through the skip-layer mechanism but also ensures the accuracy and compliance of financial information through iterative verification, meeting the stringent standards of risk control and regulatory requirements in the fintech field.

[0081] In one embodiment, a model inference decoding apparatus based on dynamic layer skipping is provided, which corresponds one-to-one with the model inference decoding method based on dynamic layer skipping described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the model inference and decoding device based on dynamic layer skipping of the present invention. The modules include: draft model initialization module 10, layer skipping mode module 20, optimal layer skipping module 30, draft model update module 40, candidate token module 50, and model decoding module 60. Detailed descriptions of each functional module are as follows: The draft model initialization module 10 is used to pre-build the target large language model and define the draft model through the target large language model. Skip-level mode module 20 is used to obtain several skip-level schemes and construct a skip-level mode set based on the skip-level schemes; The optimal skip module 30 is used to select the optimal skip pattern from the skip pattern set based on the current input state and the layer contribution of the target large language model. The draft model update module 40 is used to apply the optimal skipping pattern to the target large language model and update the draft model according to the optimal skipping pattern. The candidate token module 50 is used to perform G-step autoregressive inference through the updated draft model to obtain a number of candidate tokens and determine the number of acceptable tokens based on the number of candidate tokens. The model decoding module 60 is used to perform text decoding of the target large language model starting from a preset starting position according to the number of acceptable tokens.

[0082] In one embodiment, the draft model initialization module 10 includes: Pre-construct a large target language model; The discontinuous subset layer of the target large language model is defined as a draft model; The number of layers in the draft model is less than the total number of layers in the target large language model.

[0083] In one embodiment, the skip-level mode module 20 includes: Design several sets of differentiated skipping schemes, each set of skipping schemes corresponding to the skipping rules of each layer in the target large language model; The skipping rules include skipping even-numbered layers of the target large language model, skipping consecutive or discrete layers with preset ordinal numbers in the target large language model; By integrating the aforementioned group of skip-level schemes, a skip-level pattern set is constructed; Each element in the skip pattern set corresponds to an independent skip scheme.

[0084] In one embodiment, the optimal skip-layer module 30 includes: During the decoding process, the activation sparsity of the attention matrix of each layer of the target large language model is obtained; The contribution of the corresponding layer in the current context is quantified based on the activated sparse metric; Based on the contribution level and the current input state, the optimal skipping mode is selected from the skipping mode set.

[0085] In one embodiment, the draft model update module 40 includes: The optimal skipping pattern is applied to the target large language model; Skip redundant layers of the target large language model according to the optimal skipping pattern; The non-skipped layer after performing a skip layer in the target large language model is defined as the draft model.

[0086] In one embodiment, the candidate token module 50 includes: Perform G-step autoregressive inference using the updated draft model to generate several candidate tokens; The candidate tokens are input into the target large language model, and the candidate tokens are verified in parallel by the target large language model to obtain the verification result of each candidate token. Based on the verification results, calculate the generation probability of the target large language model for each candidate token and the generation probability of the draft model for each candidate token. Based on the standard speculative decoding algorithm, calculate the ratio of the generation probability of the target large language model for each candidate token to the generation probability of the draft model for each candidate token; The validity of each candidate token is determined based on the ratio, and the number of acceptable tokens is counted based on the validity.

[0087] In one embodiment, the model decoding module 60 includes: Set the acceptable token count to ; Set the initial token position before executing the G-step autoregressive inference to t; Update the token position of the current inference progress to the [number]th [item]. The token, i.e., the first Each token is the preset starting position for text decoding; From the Starting from the token position, the process of filtering the optimal skipping mode based on the current input state, updating the draft model, generating candidate tokens for verification, and determining the number of acceptable tokens is repeated; Repeat the above process until the target large language model completes text decoding.

[0088] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a dynamic layer-hopping model speculation decoding method on the server side.

[0089] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the user-side functions or steps of a model speculation decoding method based on dynamic skipping layers. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: A target large language model is pre-constructed, and a draft model is defined using the target large language model; Obtain several skipping schemes, and construct a skipping pattern set based on the skipping schemes; Based on the current input state and the layer contribution of the target large language model, the optimal skipping pattern is selected from the skipping pattern set; The optimal skipping pattern is applied to the target large language model, and the draft model is updated according to the optimal skipping pattern. The updated draft model is used to perform G-step autoregressive inference to obtain several candidate tokens, and the number of acceptable tokens is determined based on these candidate tokens. The target large language model performs text decoding starting from a preset starting position based on the number of acceptable tokens.

[0090] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: A target large language model is pre-constructed, and a draft model is defined using the target large language model; Obtain several skipping schemes, and construct a skipping pattern set based on the skipping schemes; Based on the current input state and the layer contribution of the target large language model, the optimal skipping pattern is selected from the skipping pattern set; The optimal skipping pattern is applied to the target large language model, and the draft model is updated according to the optimal skipping pattern. The updated draft model is used to perform G-step autoregressive inference to obtain several candidate tokens, and the number of acceptable tokens is determined based on these candidate tokens. The target large language model performs text decoding starting from a preset starting position based on the number of acceptable tokens.

[0091] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0092] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0094] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A model inference decoding method based on dynamic layer skipping, characterized in that, Includes the following steps: A target large language model is pre-constructed, and a draft model is defined using the target large language model; Obtain several skipping schemes, and construct a skipping pattern set based on the skipping schemes; Based on the current input state and the layer contribution of the target large language model, the optimal skipping pattern is selected from the skipping pattern set; The optimal skipping pattern is applied to the target large language model, and the draft model is updated according to the optimal skipping pattern. The updated draft model is used to perform G-step autoregressive inference to obtain several candidate tokens, and the number of acceptable tokens is determined based on these candidate tokens. The target large language model performs text decoding starting from a preset starting position based on the number of acceptable tokens.

2. The model inference and decoding method based on dynamic layer skipping as described in claim 1, characterized in that, The pre-construction of the target large language model, and the definition of the draft model through the target large language model, includes: Pre-construct a large target language model; The discontinuous subset layer of the target large language model is defined as a draft model; The number of layers in the draft model is less than the total number of layers in the target large language model.

3. The model inference decoding method based on dynamic layer skipping as described in claim 1, characterized in that, The step of acquiring several skip-level schemes and constructing a skip-level pattern set based on the skip-level schemes includes: Design several sets of differentiated skipping schemes, each set of skipping schemes corresponding to the skipping rules of each layer in the target large language model; The aforementioned skipping schemes are integrated to construct a skipping pattern set; wherein each element in the skipping pattern set corresponds to an independent skipping scheme.

4. The model inference and decoding method based on dynamic layer skipping as described in claim 1, characterized in that, The step of selecting the optimal skipping pattern from the skipping pattern set based on the current input state and the layer contribution of the target large language model includes: During the decoding process, the activation sparsity of the attention matrix of each layer of the target large language model is obtained; The contribution of the corresponding layer in the current context is quantified based on the activated sparse metric; Based on the contribution level and the current input state, the optimal skipping mode is selected from the skipping mode set.

5. The model inference decoding method based on dynamic layer skipping as described in claim 1, characterized in that, The step of applying the optimal skipping pattern to the target large language model and updating the draft model according to the optimal skipping pattern includes: The optimal skipping pattern is applied to the target large language model; Skip redundant layers of the target large language model according to the optimal skipping pattern; The non-skipped layer after performing a skip layer in the target large language model is defined as the draft model.

6. The model inference decoding method based on dynamic layer skipping as described in claim 1, characterized in that, The step of performing G-step autoregressive inference using the updated draft model to obtain several candidate tokens, and determining the acceptable number of tokens based on these candidate tokens, includes: Perform G-step autoregressive inference using the updated draft model to generate several candidate tokens; The candidate tokens are input into the target large language model, and the candidate tokens are verified in parallel by the target large language model to obtain the verification result of each candidate token. Based on the verification results, calculate the generation probability of the target large language model for each candidate token and the generation probability of the draft model for each candidate token. Based on the standard speculative decoding algorithm, calculate the ratio of the generation probability of the target large language model for each candidate token to the generation probability of the draft model for each candidate token; The validity of each candidate token is determined based on the ratio, and the number of acceptable tokens is counted based on the validity.

7. The model inference decoding method based on dynamic layer skipping as described in claim 1, characterized in that, The target large language model performs text decoding starting from a preset starting position based on the acceptable number of tokens, including: Set the acceptable token count to ; Set the initial token position before executing the G-step autoregressive inference to t; Update the token position of the current inference progress to the [number]th [item]. The token, i.e., the first Each token is the preset starting position for text decoding; From the Starting from the token position, the process of filtering the optimal skipping mode based on the current input state, updating the draft model, generating candidate tokens for verification, and determining the number of acceptable tokens is repeated; Repeat the above process until the target large language model completes text decoding.

8. A model inference decoding device based on dynamic layer skipping, characterized in that, The model inference decoding device based on dynamic layer skipping includes: The initial module for the draft model is used to pre-build the target large language model and define the draft model through the target large language model; The skip-layer mode module is used to obtain several skip-layer schemes and construct a skip-layer mode set based on the skip-layer schemes; The optimal skip module is used to select the optimal skip pattern from the skip pattern set based on the current input state and the layer contribution of the target large language model. The draft model update module is used to apply the optimal skipping pattern to the target large language model and update the draft model according to the optimal skipping pattern. The candidate token module is used to perform G-step autoregressive inference through the updated draft model to obtain several candidate tokens, and to determine the number of acceptable tokens based on the several candidate tokens; The model decoding module is used to perform text decoding of the target large language model starting from a preset starting position according to the number of acceptable tokens.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a model speculation decoding program based on dynamic layer skipping stored in the memory and executable on the processor. When executed by the processor, the model speculation decoding program based on dynamic layer skipping implements the steps of the model speculation decoding method based on dynamic layer skipping as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a model speculation decoding program based on dynamic layer skipping, which, when executed by a processor, implements the steps of the model speculation decoding method based on dynamic layer skipping as described in any one of claims 1-7.

Images