Model inference method, apparatus, device, and storage medium

Abstract

The application discloses a model inference method and device, equipment and storage medium, relates to the technical field of large model application, and comprises the following steps: obtaining regular training data suitable for an adaptive data synthesis scene, a target model to be deployed and a draft model; performing parallel training on the draft model according to the regular training data and a preset parallel decoding strategy to obtain a parallel draft model; modifying the source code of an inference framework suitable for a domestic graphics card, inserting parallel draft decoding logic, and obtaining a customized inference framework supporting parallel draft inference; and based on the customized inference framework, jointly inferring the target model and the parallel draft model, accelerating the processing of an inference request to be sent, and completing effect verification. The application jointly infers by constructing a parallel draft model and adapting a domestic graphics card inference framework, effectively alleviates the problem of shortage of domestic graphics card computing resources, overcomes the defects of insufficient adaptation of an existing inference framework and high inference delay, and significantly improves the model inference efficiency.

Description

Technical Field

[0001] This application relates to the field of large model application technology, and in particular to a model reasoning method, apparatus, device and storage medium. Background Technology

[0002] With the rapid popularization of artificial intelligence technology and large-scale model applications, the demand for model inference efficiency in scenarios such as data intelligent synthesis continues to increase. Inference acceleration has become a key technology direction for the implementation of large models, while the construction and adaptation optimization of the domestic hardware ecosystem has also become an important trend in industry development. Domestic graphics cards are gradually taking on the core computing power support role in large-scale model inference tasks, and technologies such as multi-card parallelism and model splitting have been used to improve inference efficiency.

[0003] Currently, draft model inference is one of the mainstream solutions for improving decoding speed in large-scale model inference acceleration technologies. Traditional draft models optimize the inference process by generating candidate tokens and having them verified in parallel by the target model. Meanwhile, inference frameworks such as VLLM (V-Virtual Large Language Model) provide basic support for efficient inference of large models, and related technologies have been initially applied and verified on general computing platforms. However, existing draft model inference still adopts an autoregressive generation method, which can only generate a single token per forward pass, resulting in significant inference latency. Furthermore, the related acceleration algorithms and frameworks are not well adapted to the ecosystem of domestic graphics cards such as Huawei Ascend, facing issues such as missing documentation, incomplete toolchains, and poor compatibility and stability. At the same time, domestic graphics cards generally suffer from a shortage of computing resources, making it difficult to meet the high-efficiency, low-latency inference requirements of data intelligent synthesis scenarios, thus restricting the stable implementation of model training data processing under resource constraints. Summary of the Invention

[0004] The main purpose of this application is to provide a model inference method, apparatus, device and storage medium, which aims to effectively alleviate the shortage of computing power resources of domestic graphics cards, overcome the shortcomings of insufficient adaptation of existing inference frameworks and high inference latency, and significantly improve the model inference efficiency in data synthesis scenarios.

[0005] To achieve the above objectives, this application proposes a model reasoning method, the method comprising: Acquire well-organized training data suitable for the data synthesis scenario, as well as the target model and draft model to be deployed; Based on the regularized training data and the preset parallel decoding strategy, the draft model is trained in parallel to obtain a parallel draft model that can generate multiple candidate text units at once. The source code of the inference framework adapted to domestic graphics cards was modified, and parallel draft decoding logic was inserted to obtain a customized inference framework that supports parallel draft inference. Based on the customized inference framework, the target model and the parallel draft model are jointly inferred to accelerate the inference request to be sent and to verify the effect.

[0006] In one possible implementation, acquiring the regularized training data adapted to the data synthesis scenario, as well as the target model and draft model to be deployed, includes: Obtain the original question-and-answer text data in the data synthesis scenario described above; The original question-and-answer text data is cleaned and format-converted to obtain well-organized training data suitable for model training. Based on the regularized training data, a target model for executing the final output and a draft model for generating candidate content are determined.

[0007] In one possible implementation, the step of training the draft model in parallel based on the regularized training data and a preset parallel decoding strategy to obtain a parallel draft model capable of generating multiple candidate text units at once includes: Based on the regularized training data, a multi-step parallel prediction task is constructed; Based on the preset parallel decoding strategy, a mask placeholder structure and a conditional discarding mechanism are introduced; The draft model is trained based on the mask placeholder structure and the conditional discarding mechanism to obtain a parallel draft model that can generate multiple candidate text units at once.

[0008] In one possible implementation, training the draft model based on the mask placeholder structure and the conditional discarding mechanism includes: The training data is divided into independent parallel subtasks based on the regularized training data; The mask placeholder structure is used to shield future information during the training process in order to maintain consistency between training and inference. The conditional discarding mechanism reduces training overhead and maintains the integrity of contextual information, enabling iterative parallel training of the draft model.

[0009] In one possible implementation, the source code of the inference framework adapted to domestic graphics cards is modified, and parallel draft decoding logic is inserted to obtain a customized inference framework that supports parallel draft inference, including: Obtain the source code of the native inference framework adapted to the domestic graphics card environment; Add decoding call logic for parallel draft inference tasks to the decoding execution flow of the original inference framework; Based on the added decoding call logic, compatibility adaptation with domestic hardware environments was completed, resulting in a customized inference framework that supports parallel draft inference.

[0010] In one possible implementation, the step of jointly reasoning the target model and the parallel draft model based on the customized inference framework to accelerate the inference request to be sent and complete the effect verification includes: Obtain the inference request to be sent and the inference parameters corresponding to the inference request; Multiple candidate text units are generated through the parallel draft model, and the target model performs parallel verification and adoption to complete the inference output and statistical inference performance indicators based on the verification and adoption results. Based on the aforementioned inference performance metrics, the inference acceleration effect and content consistency verification were completed.

[0011] In one possible implementation, the step of verifying the inference acceleration effect and content consistency based on the inference performance metrics includes: The generation efficiency and time consumption indicators in the statistical reasoning process are analyzed, and the acceleration effect is calculated based on the generation efficiency and time consumption indicators. Compare the inference output before and after acceleration to determine the consistency of the content; The overall solution was verified based on the acceleration effect and the consistency of the content.

[0012] Furthermore, to achieve the above objectives, this application also proposes a model reasoning device, which includes: The acquisition module is used to acquire the regularized training data adapted to the data synthesis scenario, as well as the target model and draft model to be deployed. The training module is used to train the draft model in parallel based on the regularized training data and the preset parallel decoding strategy, so as to obtain a parallel draft model that can generate multiple candidate text units at one time. The adaptation module is used to modify the source code of the inference framework adapted to domestic graphics cards, insert parallel draft decoding logic, and obtain a customized inference framework that supports parallel draft inference. The inference module is used to perform joint inference between the target model and the parallel draft model based on the customized inference framework, accelerate the inference request to be sent, and complete the effect verification.

[0013] In addition, to achieve the above objectives, this application also proposes a model inference device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the model inference method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the model inference method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the model reasoning method described above.

[0016] This application provides a model inference method, apparatus, device, and storage medium. The model inference method acquires regularized training data adapted to a data synthesis scenario, as well as a target model and a draft model to be deployed. Then, based on the regularized training data and a preset parallel decoding strategy, the draft model is trained in parallel to obtain a parallel draft model that can generate multiple candidate text units at once. The source code of the inference framework adapted to domestic graphics cards is modified to insert parallel draft decoding logic, resulting in a customized inference framework that supports parallel draft inference. Based on the customized inference framework, the target model and the parallel draft model are jointly inferred to accelerate the inference request to be sent and complete the effect verification. By constructing a parallel draft model and adapting it to the inference framework of domestic graphics cards for joint inference, the problem of scarce computing power resources of domestic graphics cards is effectively alleviated, the defects of insufficient adaptation and high inference latency of existing inference frameworks are overcome, and the model inference efficiency in data synthesis scenarios is significantly improved. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the model reasoning method in this application. Figure 2 A schematic diagram illustrating the principle of traditional draft model reasoning provided for the model reasoning method of this application; Figure 3 A schematic diagram illustrating the principle of parallel draft model inference provided for the model inference method of this application; Figure 4This is a schematic diagram of the device structure of the hardware operating environment involved in the model inference method in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, big data service platform, or model inference system capable of realizing the above functions. The following description uses a model inference system as an example to illustrate this embodiment and the subsequent embodiments.

[0024] Based on this, embodiments of this application provide a model reasoning method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the model reasoning method in this application.

[0025] In this embodiment, the model inference method includes steps S11 to S14: Step S11: Obtain the regularized training data adapted to the data synthesis scenario, as well as the target model and draft model to be deployed. It should be noted that the data synthesis scenario refers to the intelligent generation and processing scenario for samples required for model training; the regularized training data refers to a high-quality question-and-answer dataset that can be directly used for model fine-tuning after cleaning, deduplication, and format standardization; the target model refers to a large-parameter model that is used to undertake the final inference output and has high generation quality and stability; and the draft model refers to a small-parameter model that is used to quickly generate candidate content and help improve inference efficiency.

[0026] In one possible implementation, the system collects domain-related raw data through a business interface and processes it to obtain regularized training data. In another possible implementation, the system selects a target model and a draft model with corresponding parameter quantities based on task complexity and hardware capacity.

[0027] Specifically, based on the domain requirements of data intelligent synthesis services, the system collects and processes standardized training data with uniform format and valid content. Simultaneously, based on the scale of the training data, domain complexity, and the computing power of domestically produced hardware, it selects the target model for executing the final inference output and a draft model to assist in generating candidate content, completing the initialization preparation of the data and models. For example, the system acquires standardized training data from the mathematics and knowledge question answering domains, selecting a model with 1.4 billion parameters as the target model and a model with 400 million parameters as the draft model to adapt to subsequent parallel training and inference acceleration processes.

[0028] Step S12: Based on the regularized training data and the preset parallel decoding strategy, the draft model is trained in parallel to obtain a parallel draft model that can generate multiple candidate text units at one time. It should be noted that the preset parallel decoding strategy refers to the parallel computation and decoding execution rules used to enable the model to generate multiple text units in a single forward pass. Candidate text units refer to the basic text components (tokens) generated by the draft model and provided to the target model for verification and adoption. Parallel training refers to the model optimization method that transforms traditional single-step prediction into multi-task synchronous prediction. The parallel draft model refers to the optimized draft model that, after parallel training, has the ability to generate multiple candidate text units at once. In this embodiment, the system performs parallel training on the draft model based on regularized training data and the preset parallel decoding strategy. This breaks the sequential dependency of traditional autoregressive generation, reduces the time consumed by draft generation during the inference stage, and improves overall decoding efficiency. In one possible implementation, the system uses a mask placeholder method to achieve decoupled parallel generation between text units. In another possible implementation, the system uses a conditional discarding mechanism to control the amount of training data to reduce training overhead.

[0029] Specifically, the system inputs regularized training data into the draft model, constructs a multi-task parallel prediction structure according to a preset parallel decoding strategy, ensures training stability through masking placeholders and conditional discarding, and through multiple rounds of iterative optimization, enables the draft model to generate multiple candidate text units at once, ultimately resulting in a parallel draft model suitable for inference acceleration. For example, the system performs parallel training on a draft model with 400 million parameters based on regularized training data, enabling the model to generate multiple candidate text units at once, significantly reducing the draft generation time during the inference stage.

[0030] Step S13: Modify the source code of the inference framework adapted to domestic graphics cards, insert parallel draft decoding logic, and obtain a customized inference framework that supports parallel draft inference. It should be noted that "domestic graphics card" refers to a domestically produced hardware acceleration chip with independent computing power support; "inference framework" refers to the software runtime environment used to load models and perform inference calculations; "parallel draft decoding logic" refers to the code flow used to call the parallel draft model to complete the generation and verification of multiple candidate text units; and "customized inference framework" refers to a dedicated inference environment that has been modified and adapted to parallel draft inference and domestic hardware. In this embodiment, the system modifies the code of the native inference framework adapted to domestic graphics cards, inserts decoding execution logic adapted to the parallel draft model, makes up for the deficiency of the native framework not supporting parallel draft inference, and realizes the compatibility and functional expansion of the inference framework under the domestic hardware environment.

[0031] Specifically, the system obtains the source code of the native inference framework adapted to domestic graphics cards, inserts program logic for parallel draft model calls, candidate text unit generation, result return and verification into the core decoding execution process, and then completes compatibility debugging based on the computing power scheduling, memory management and operator support characteristics of domestic hardware. Finally, a customized inference framework that can stably support parallel draft inference is obtained. This framework is fully compatible with the model format and encoding method in the training phase, realizing seamless connection between training and inference.

[0032] For example, the system modifies the source code of the vllm-ascend 0.9.1rc3 inference framework adapted to Huawei Ascend graphics cards, adds parallel draft inference decoding logic and completes hardware adaptation, resulting in a customized inference framework that can run on domestic graphics cards.

[0033] Step S14: Based on the customized inference framework, the target model and the parallel draft model are jointly inferred to accelerate the inference request to be sent and complete the effect verification.

[0034] It should be noted that joint inference refers to a reasoning mode in which the parallel draft model and the target model work together, with the draft model generating first and the target model then verifying; the inference request to be sent refers to user questions or sample generation requests from the data synthesis scenario; accelerated processing refers to reducing inference latency and increasing the speed of text unit generation through parallel generation; and effect verification refers to detecting and confirming the inference acceleration ratio and the consistency of output content. In this embodiment, the system relies on a customized inference framework to schedule the target model and the parallel draft model to work together, complete the rapid response to inference requests, and verify the acceleration effect through performance statistics and content comparison, ensuring that it can still stably support data synthesis business even in environments with scarce computing power. In one possible implementation, the system generates multiple candidate text units from the parallel draft model, and the target model batch verifies and adopts the valid results; in another possible implementation, the system calculates indicators such as the number of text units generated per second and the total time to complete the effect evaluation.

[0035] Specifically, in this embodiment, the system fully utilizes the computing power of domestically produced graphics cards through joint inference, improving inference efficiency while maintaining the same generation quality. For example, the system schedules a target model with 1.4 billion parameters and a parallel draft model with 400 million parameters to perform joint inference based on a customized inference framework. It processes 1,000 data synthesis inference requests, with the parallel draft model generating four candidate text units at a time. The target model completes verification and adoption, ultimately resulting in a 15% improvement in inference efficiency, and the output content of the models before and after acceleration is completely consistent.

[0036] This embodiment obtains regularized training data and corresponding models adapted to the data synthesis scenario, trains a parallel draft model with parallel generation capabilities, modifies the inference framework to adapt to domestic graphics cards, and then completes request acceleration and effect verification through dual-model joint inference. This can effectively alleviate the shortage of computing power resources of domestic graphics cards, reduce inference latency, and improve the model inference efficiency in data synthesis scenarios.

[0037] Understandably, traditional draft models typically employ an autoregressive generation approach, generating candidate text units one by one, which are then validated in parallel by the target model. While this can improve inference efficiency to some extent, it still suffers from significant latency in the draft inference stage, making it difficult to meet the actual needs of efficient inference in data synthesis scenarios. (See reference...) Figure 2 The AR Draft model adopts an autoregressive draft generation mechanism. Its core is to generate candidate token sequences word by word through the AR Draft Model. Each step outputs tokens such as "story", "about", and "large" in sequence based on the generated preceding content. The complete candidate sequence is then input into the Target Model for verification and correction. This method follows the dependency relationship of traditional autoregressive generation, and there is a significant sequential delay in the generation process.

[0038] To address the aforementioned issues, this application adapts the traditional autoregressive draft model to a parallel draft model, breaking the sequential dependency of generation and enabling the generation of multiple candidate text units in a single forward propagation. This fundamentally reduces the draft generation time. (See reference...) Figure 3The PARD model employs a parallel draft generation mechanism. It processes the input context in parallel using a ParaDraft Model, simultaneously generating multiple candidate tokens with placeholder masks. This allows for the construction of batch candidate sequences without relying on previously generated tokens. The parallel-generated candidate results are then uniformly fed into the Target Model for verification, eliminating the step dependencies inherent in autoregressive generation and improving the overall efficiency of the draft generation stage. During model training, this application introduces a conditional discarding text unit training strategy. Training text units are selectively filtered and retained according to preset rules. This maintains the integrity of contextual information, ensures the effectiveness of the cache structure, reduces the total amount of training data and computational overhead, significantly reducing draft time during inference and thus improving overall decoding speed.

[0039] In other words, traditional draft models generate K tokens autoregressively, requiring K forward propagations. Parallel draft modeling introduces mask tokens as placeholders, breaking the dependencies between tokens and achieving K token generation in a single forward propagation. Parallel training significantly increases the number of training tokens. A conditional token discarding training strategy is adopted, introducing a geometrically decaying retention rate r, retaining tokens for the i-th subtask. Tokens, minimum retention rate The retained tokens still provide a complete KVcache, ensuring context consistency and reducing the total number of training tokens from [previous value]. Reduce to near This significantly reduces training costs.

[0040] In summary, traditional AR models predict only the next token at each step during training, and the loss is calculated solely based on the prediction result of the current step. The parallel draft model divides the training data into K independent subtasks, each responsible for predicting future tokens at different positions. At each step, it simultaneously predicts the next K tokens, and the loss is the sum of the losses from multiple subtasks. Furthermore, a mask token is used to mask future information, ensuring consistency between training and inference.

[0041] In one embodiment, the inference performance of the parallel draft model was verified in an intelligent data synthesis business scenario. Synthetic training data was used, and the Qwen3 series models were selected for training and testing. Considering the requirements of the intelligent data synthesis scenario on model capabilities and the inference acceleration efficiency of the draft model, the platform selected was: target model Qwen3-14B-Instruct, and draft models Qwen3-0.6B-Instruct and Qwen3-4B-Instruct; the number of parallel inference tokens for training the draft models (i.e., the number of tokens generated in one decoding operation) was 4 and 8, respectively. Fine-tuning of all Qwen3 models was performed using four Huawei 910b cards for 5 epochs, with an initial learning rate of 3e-5.

[0042] Additionally, the native vllm and vllm-ascend-0.9.1 do not support parallel draft model inference prediction, requiring the addition of parallel decoding logic to the source code. Specifically, the vllm-ascend adaptation requires adding a call to the parallel draft inference decoding process during the decoding process in patch_multi_step_workser.py.

[0043] Furthermore, the parallel draft inference acceleration effect of the model was tested on TensorFlow (TF). After fine-tuning and training on multiple models in the Qwen3 series, the platform randomly selected 1000 synthetic test data. During the test, the batch_size was set to 1, and the total request time and the number of tokens generated were recorded. The number of parallel processes and the draft acceptance rate of different model sizes were compared. The experimental results are shown in the table below. With a parallel process count of 8, the Qwen3-4B parallel draft model can achieve an acceptance rate of 3 and an inference speedup of 2.03X, reducing the inference time of the intelligent synthetic model by more than half. See Table 1 for details. Table 1 represents the inference acceleration effect of the parallel draft model transformer.

[0044] Table 1

[0045] Furthermore, to simulate real-world usage scenarios, the platform prepared 1000 data points and divided them into 100 data requests using a non-random backsampling method. Each request data list had a length between 1 and 50 characters, simulating both random and batch requests. The Qwen3-4B-sft model (with 8 parallel iterations) was selected as the draft model, and the intelligent synthesis model as the target model. The test results are shown in Table 2, which represents the parallel draft model VLLM inference test. Table 2

[0046] Through experimental comparison, since the maximum acceptance rate of the draft model is 3, when the inference parameter num_speculative_tokens=4 is set in the draft model, the model can reduce the inference time by 139 seconds compared to the original inference VLLM inference, and the speedup ratio can reach 1.15 times, which is consistent with the acceptance rate of the draft model. After verifying the inference results before and after the acceleration, the model generates consistent results with basically no difference in content.

[0047] In summary, through the above-mentioned model structure improvements, training strategy optimizations, and inference process reconstruction, this application can effectively overcome the shortcomings of existing technologies such as insufficient adaptation to domestic graphics card environments, high latency, and limited computing resources, and meet the needs of efficient and stable inference in data intelligent synthesis scenarios under resource constraints.

[0048] In one feasible implementation, acquiring the regularized training data adapted to the data synthesis scenario, as well as the target model and draft model to be deployed, includes: Step S21: Obtain the original question-and-answer text data in the data synthesis scenario; It should be noted that raw question-and-answer text data refers to user questions and corresponding responses collected directly from the business environment without any processing. In this embodiment, the system acquires raw question-and-answer text data that aligns with business needs, ensuring that model training remains consistent with the business scenario. In one possible implementation, the system collects raw question-and-answer text data through a business data interface; in another possible implementation, the system obtains raw question-and-answer text data through extraction from a business database.

[0049] Specifically, based on the business scope and domain type of the data synthesis scenario, the system filters and extracts user questions and standard answers for the corresponding domain, forming a raw text set containing multiple question-and-answer pairs. This ensures that the raw question-and-answer text data covers the knowledge scope and question types required by the business. For example, the system obtains raw question-and-answer text data containing mathematical problem solving and knowledge-based question answering from the data intelligence synthesis business, forming the initial training data source.

[0050] In one embodiment, in a data synthesis scenario, 20,000 pieces of model training data for the data synthesis scenario are prepared. The original data consists of standard QA pairs of text. Data examples are as follows: {"user": "How to solve the equation x^2 = 4?", "Assistant": "First, take the square root of both sides of the equation. This gives x = ±2. Therefore, the solution is 2 or -2."} Step S22: Clean and convert the original question and answer text data to obtain well-organized training data suitable for model training. It should be noted that cleaning refers to the processing operations of deduplication, error correction, and removal of invalid content from the original question and answer text data, while format conversion refers to converting non-standardized question and answer text into a unified dialogue format that the model can read.

[0051] Specifically, the system first removes duplicate content, corrects errors, and filters invalid statements from the original question-and-answer text data. Then, it transforms the cleaned data into a standardized dialogue structure containing system prompts, user questions, and assistant responses, according to model training requirements. This results in well-organized training data that meets the training input requirements. For example, the system removes duplicates and corrects errors from the original mathematical question-and-answer text and converts it into a unified dialogue template format, obtaining well-organized training data that can be directly used for model fine-tuning.

[0052] Step S23: Based on the regularized training data, determine the target model for executing the final output and the draft model for generating candidate content.

[0053] It should be noted that the target model refers to the high-parameter model used to complete the final inference and output stable results, while the draft model refers to the low-parameter model used to quickly generate candidate content to assist in accelerating inference. Candidate content refers to the text content generated by the draft model and then verified and adopted by the target model. In this embodiment, the system matches corresponding model combinations based on the regularized training data to ensure that the model capabilities are adapted to the task requirements. In one possible implementation, the system selects models with different parameter counts based on data complexity; in another possible implementation, the system determines the model combination based on hardware resources.

[0054] Specifically, based on the domain complexity, data scale, and inference efficiency requirements of the regularized training data, the system selects a model with strong generation capabilities and stability as the target model, and simultaneously selects a model with fast computation speed and low resource consumption as the draft model, enabling the two to work together to accelerate inference in subsequent processes. For example, based on the domain complexity, data scale, and inference efficiency requirements of the regularized training data, the system selects a model with strong generation capabilities and high stability as the target model (using Qwen3-14B, 1.4 billion parameters), and selects a model with fast computation speed and low resource consumption as the draft model (using Qwen3-4B, 400 million parameters), forming a model combination suitable for subsequent acceleration processes, while ensuring that both use a unified encoding format and data processing flow.

[0055] This embodiment obtains raw question-and-answer text data in a data synthesis scenario, performs cleaning and format conversion on the raw data to obtain regularized training data, and then matches the target model and draft model based on the regularized training data, providing a reliable data foundation and a suitable model carrier for subsequent model training and inference acceleration, ensuring the stable implementation of the overall solution under resource constraints.

[0056] In one feasible implementation, the step of training the draft model in parallel based on the regularized training data and a preset parallel decoding strategy to obtain a parallel draft model capable of generating multiple candidate text units at once includes: Step S31: Based on the regularized training data, construct a multi-step parallel prediction task; It should be noted that the multi-step parallel prediction task refers to the training task of transforming the model's single prediction of a single text unit into a single synchronous prediction of multiple subsequent text units. In this embodiment, the system constructs a multi-step parallel prediction task based on regularized training data to break the sequential dependency of traditional autoregressive generation and provide a task foundation for the parallel generation of the draft model. In one possible implementation, the system divides the prediction target according to a set number of parallel steps; in another possible implementation, the system splits the corresponding multi-step prediction content according to the text length.

[0057] Specifically, based on regularized training data, the system divides each piece of text data into multiple synchronous prediction sub-objectives according to a preset parallel quantity. This allows the draft model to simultaneously learn prediction rules for text units at multiple subsequent positions during the same forward process, forming a multi-step parallel prediction task that can support parallel generation. For example, based on regularized training data in the field of mathematics, the system transforms a single-step prediction task into a multi-step parallel prediction task that synchronously predicts four subsequent text units, which is then used for the parallel training of the draft model.

[0058] Step S32: According to the preset parallel decoding strategy, a mask placeholder structure and a conditional discarding mechanism are introduced. It should be noted that the preset parallel decoding strategy refers to the decoding rules and computational logic used to achieve parallel generation of the draft model; the mask placeholder structure refers to the structure that uses placeholder symbols to shield future information and ensure consistency between training and inference; and the conditional discarding mechanism refers to the processing method that selectively retains training text units according to rules to reduce training overhead. In this embodiment, the system introduces the corresponding structure and mechanism based on the preset parallel decoding strategy to ensure the stability of parallel training and reduce resource consumption. In one possible implementation, the system uses attention masks to construct the mask placeholder structure; in another possible implementation, the system uses decay retention rules to implement conditional discarding.

[0059] Specifically, the system adds mask placeholders to the model input according to a preset parallel decoding strategy to shield future information. Simultaneously, it selectively retains and filters training text units according to predefined retention rules, reducing the amount of training data and computational overhead while ensuring contextual integrity. For example, the system adds mask placeholders to the training input according to the parallel decoding strategy and uses a geometric decay rule to selectively retain training text units, thus introducing the mask placeholder structure and conditional discarding mechanism.

[0060] Step S33: The draft model is trained based on the mask placeholder structure and the conditional discarding mechanism to obtain a parallel draft model that can generate multiple candidate text units at once.

[0061] It should be noted that in this embodiment, the system trains the draft model using a mask placeholder structure and a conditional discarding mechanism as constraints, enabling the model to have parallel generation capabilities and providing core support for accelerating subsequent joint inference. In one possible implementation, the system uses supervised fine-tuning to complete model training; in another possible implementation, the system completes training optimization based on a multi-subtask loss summation method.

[0062] Specifically, the system inputs regularized training data into the draft model, uses a mask placeholder structure to ensure information masking, and employs a conditional discarding mechanism to control training overhead. Through multiple rounds of iterative optimization of model parameters, the model is able to generate multiple candidate text units in a single forward pass, ultimately resulting in a parallel draft model. For example, the system performs supervised fine-tuning training on a draft model with 400 million parameters based on the mask placeholder structure and conditional discarding mechanism, enabling the model to generate multiple candidate text units at once, thus obtaining a parallel draft model suitable for inference acceleration.

[0063] This embodiment constructs a multi-step parallel prediction task based on regularized training data, introduces a mask placeholder structure and a conditional discarding mechanism according to the parallel decoding strategy, and then trains the draft model to obtain a parallel draft model that can generate multiple text units in parallel. This effectively reduces the model training overhead, improves the draft generation efficiency, and provides key capability support for inference acceleration in the subsequent domestic environment.

[0064] In one feasible implementation, training the draft model based on the mask placeholder structure and the conditional discarding mechanism includes: Step S41: Divide the regularized training data into independent parallel subtasks; It should be noted that independent parallel subtasks refer to breaking down the overall prediction target into multiple independent, synchronously computeable text unit prediction subtasks. In this embodiment, the system divides independent parallel subtasks based on regularized training data, allowing the draft model to process multiple prediction targets simultaneously in the same round of forward computation, providing task support for parallel generation. In one possible implementation, the system equally divides the overall prediction target according to a preset number of parallel operations; in another possible implementation, the system divides independent parallel subtasks according to text semantic segments.

[0065] Specifically, the system breaks down the overall prediction target corresponding to the regularized training data into multiple independent and synchronously executable text unit prediction subtasks according to a set number of parallel generation steps. This allows the draft model to learn and predict text units at multiple future positions simultaneously. For example, based on regularized training data from the mathematical domain, the system breaks down a single prediction target into four independent parallel subtasks, each corresponding to the prediction of subsequent text units at different positions.

[0066] Step S42: Based on the mask placeholder structure, future information during the training process is shielded to maintain consistency between training and inference; It should be noted that the mask placeholder structure refers to an information masking structure constructed using placeholder symbols. Future information refers to subsequent text unit information that has not yet been generated during training and is not allowed to be perceived by the model in advance. Training-inference consistency means that the information visibility range and computational logic of the model remain the same during the training and inference phases. In this embodiment, the system uses a mask placeholder structure to mask future information, preventing the model from using illegal information during training and ensuring consistency between training behavior and actual inference behavior. In one possible implementation, the system uses attention masks to achieve information masking; in another possible implementation, the system uses placeholder symbols to mark the positions to be predicted to achieve information isolation.

[0067] Specifically, the system uses a mask placeholder structure to mark and mask subsequent ungenerated positions in the training input, ensuring that the model can only predict based on the established context during training and cannot obtain the true information of future positions, thus ensuring complete consistency between training and inference logic. For example, the system adds mask placeholder symbols to the training input to mask the positions of subsequent text units to be predicted, so that the draft model can only make parallel predictions based on the context.

[0068] Step S43: Reduce training overhead and maintain the integrity of context information according to the conditional discarding mechanism, and iteratively complete the parallel training of the draft model.

[0069] It should be noted that the conditional discarding mechanism refers to a processing mechanism that selectively retains training text units and filters redundant data according to set rules. Training overhead refers to the computational resources and time costs occupied during model training. Contextual information integrity means that the contextual semantics, positional associations, and cached information of the model input are not corrupted. In this embodiment, the system uses a conditional discarding mechanism to reduce training computation while ensuring contextual integrity, and obtains a draft model with parallel generation capabilities through multiple rounds of iterative optimization. In one possible implementation, the system uses a decay-based retention rule to perform conditional discarding; in another possible implementation, the system uses a fixed threshold method to perform conditional discarding.

[0070] Specifically, the system selectively filters training text units according to a conditional discarding mechanism. This reduces the total amount of training data and computational cost while preserving complete context cache and semantic information. Through multiple rounds of parameter iteration optimization, the system completes the parallel training process of the draft model. For example, the system conditionally discards training text units according to a geometric decay rule, reducing training overhead while maintaining context integrity. After multiple iterations, a stable parallel draft model is obtained.

[0071] This embodiment divides the training data into independent parallel subtasks, maintains consistency between training and inference based on the mask placeholder structure, and uses a conditional discarding mechanism to reduce training overhead and complete iterative model training while ensuring the integrity of the context. This enables the draft model to have stable and efficient parallel generation capabilities, while adapting to the resource constraints of domestic graphics cards and improving the overall training and inference efficiency.

[0072] In one feasible implementation, the source code modification of the inference framework adapted to domestic graphics cards, inserting parallel draft decoding logic, results in a customized inference framework supporting parallel draft inference, including: Step S51: Obtain the source code of the native inference framework adapted to the domestic graphics card environment; It should be noted that the "domestic graphics card environment" refers to the hardware operating environment equipped with domestically produced computing chips, the "native inference framework" refers to the basic software environment developed for domestic graphics cards and used to carry out model inference calculations, and the "native inference framework source code" refers to the unmodified original program code of the inference framework. In this embodiment, the system obtains the source code of the native inference framework adapted to the domestic graphics card environment, providing a basic program carrier for subsequent code modifications and functional expansions, ensuring that the framework modification can fit the operating characteristics of domestic hardware. In one possible implementation, the system obtains the source code of the native inference framework through official open-source channels; in another possible implementation, the system obtains the source code of the native inference framework through the development toolkit provided by the hardware manufacturer.

[0073] Specifically, the system matches and retrieves the original source code of the native inference framework based on the deployed domestic graphics card model and runtime environment version. This ensures that the retrieved source code is compatible with the hardware environment and model deployment requirements, providing compliant and stable basic code resources for subsequent functional modifications. For example, based on the Huawei Ascend series domestic graphics card environment, the system retrieves the complete source code of the officially adapted native inference framework as the basic program file for subsequent code modifications.

[0074] Step S52: Add decoding call logic for parallel draft inference tasks to the decoding execution flow of the native inference framework; It should be noted that the decoding execution flow refers to the core execution link of the inference framework in completing the generation and verification of text units, the parallel draft inference task refers to the inference task of calling the parallel draft model to realize the synchronous generation of multiple candidate text units, and the decoding call logic refers to the program control code used to trigger and execute the parallel draft inference task. In this embodiment, the system adds corresponding call logic to the core flow of the native inference framework, enabling the inference framework to schedule the parallel draft model to execute inference tasks, thus compensating for the functional deficiency of the native framework in not supporting parallel draft inference. In one possible implementation, the system adds parallel draft inference call logic at the beginning stage of the decoding flow and at the intermediate verification stage of the decoding flow.

[0075] Specifically, the system locates the core decoding execution node of the native inference framework and inserts program code at this node to call the parallel draft model and receive the parallel generated results. This enables the framework to automatically trigger the parallel draft inference task at runtime and connect the generated results to the subsequent verification process. For example, in the text unit decoding and verification process of the native inference framework, the system inserts the code for calling the parallel draft model and receiving the results, thus adding the decoding call logic.

[0076] Step S53: Based on the added decoding call logic, the compatibility adaptation of the domestic hardware environment is completed, resulting in a customized inference framework that supports parallel draft inference.

[0077] It should be noted that compatibility adaptation with domestic hardware environments refers to adjusting the program code to enable the inference framework to stably schedule computing power and execute computational tasks on domestic graphics cards. Customized inference frameworks refer to dedicated inference frameworks that have undergone code modification and optimization, possessing parallel draft inference capabilities. In this embodiment, the system completes hardware adaptation based on newly added decoding logic, resolving the issues of incomplete toolchains and insufficient compatibility in the domestic environment, enabling the inference framework to stably support dual-model joint inference. In one possible implementation, the system achieves compatibility adaptation by adjusting computing power scheduling parameters; in another possible implementation, the system achieves compatibility adaptation by optimizing memory allocation rules.

[0078] Specifically, the system debugs and optimizes the framework code after adding decoding call logic based on the computing power characteristics, memory management rules, and operator support capabilities of domestically produced graphics cards. This eliminates runtime conflicts and adaptation anomalies, enabling the framework to normally schedule hardware resources to execute parallel draft inference, ultimately resulting in a stable and customized inference framework. For example, the system optimizes the framework code for the computing power scheduling rules of Ascend domestic graphics cards, completes compatibility debugging, and obtains a customized inference framework that supports parallel draft inference. Therefore, to improve the speed and efficiency of model inference, the system optimizes the model's inference process through a parallel decoding framework, reducing data synthesis latency. Overcoming the shortcomings of the Huawei Ascend ecosystem, and given the lack of Huawei ecosystem documentation and an incomplete toolchain, the system independently completed the heterogeneous deployment of the inference acceleration framework, achieving performance verification in a domestic environment.

[0079] This embodiment obtains the source code of the native inference framework adapted to the domestic graphics card environment, adds the calling logic of parallel draft inference in the decoding process, and then completes the compatibility adaptation of domestic hardware to obtain a customized inference framework. This effectively solves the problem of insufficient adaptation of the native framework to domestic graphics cards and provides a stable and reliable operating environment for subsequent dual-model joint inference.

[0080] In one feasible implementation, the step of jointly reasoning the target model and the parallel draft model based on the customized inference framework to accelerate the inference request to be sent and complete the effect verification includes: Step S61: Obtain the inference request to be sent and the inference parameters corresponding to the inference request; It should be noted that the inference request to be sent refers to the business request that the model needs to respond to in the data intelligent synthesis scenario, and the inference parameters refer to the set of configuration items used to control the model inference process and determine the generation behavior and output format. In this embodiment, the system obtains the inference request to be sent and the inference parameters matching the request, which can provide clear execution basis and constraints for the subsequent joint inference process, ensuring that the inference behavior meets the business requirements and scenario settings, and at the same time ensuring that parallel draft inference and target model verification can be executed stably under a unified parameter specification. In one possible implementation, the system receives inference requests from the upper-layer application in real time through the business service interface. In another possible implementation, the system reads the cached inference requests to be processed from the local task scheduling queue and loads the associated parameters.

[0081] Specifically, the system receives text generation or question-and-answer interaction inference requests in data synthesis scenarios, and extracts inference parameters bound to the current task from the request header, task configuration, or preset configuration file. These parameters include generation length limit, sampling temperature, number of parallel candidates, tensor parallel strategy, memory usage ratio, and sequence length limit. The system then binds the request content with the control parameters to form a standardized task object that can be directly entered into the inference process.

[0082] For example, the system obtains a reasoning request to be sent containing business knowledge questions and answers, and simultaneously obtains reasoning parameters such as the maximum generation length, sampling strategy, number of candidate text units, and tensor parallel dimension corresponding to the request, providing complete execution conditions for subsequent joint reasoning.

[0083] Step S62: Multiple candidate text units are generated through the parallel draft model, and the target model performs parallel verification and adoption to complete the inference output and statistical inference performance indicators based on the verification and adoption results. It should be noted that the parallel draft model refers to an auxiliary inference model that has been trained in parallel and can generate multiple text units in a single forward pass. Candidate text units refer to the basic text units output in advance by the parallel draft model and provided to the target model for verification. The target model refers to the main inference model that undertakes the final output decision and possesses high-quality generation. Parallel verification and adoption refers to the process by which the target model simultaneously verifies multiple candidate text units, judges their legality, and retains valid content. Inference performance metrics refer to quantitative statistical data reflecting inference speed, efficiency, stability, and candidate acceptance. In this embodiment, the system rapidly provides multiple sets of candidate content through the parallel draft model, which are then uniformly verified and confirmed by the target model. This significantly reduces the number of autoregressive generation iterations and lowers the overall inference latency. Simultaneously, key metrics are collected in real-time during the inference process, providing objective data support for subsequent performance verification.

[0084] In one possible implementation, the system generates multiple consecutive candidate text units at once using a parallel draft model according to a set number of parallel processes. In another possible implementation, the system performs batch probability verification and validity determination on all candidate text units using a target model, and completes the adoption of valid units and the elimination of invalid units according to the acceptance strategy.

[0085] Specifically, the system inputs the acquired inference request into a customized inference framework. First, a parallel draft model generates multiple consecutive candidate text units based on contextual information. Then, these units are input into the target model for synchronous probability verification, semantic rationality judgment, and validity confirmation. The target model adopts valid units based on the verification results and continues the inference process based on the adopted content until completion, ultimately outputting a complete inference response text. Simultaneously, throughout the entire inference cycle, inference performance metrics such as text unit generation speed, total time consumption, single-step time consumption, number of accepted candidates, and candidate acceptance rate are collected in real time and cached and summarized. For example, the system drives the parallel draft model to generate four candidate text units at once based on the inference request. The target model synchronously verifies these units and adopts valid content, outputting the final inference text based on the adoption results. During the process, multiple inference performance metrics are statistically analyzed, including the number of text units generated per second, total time consumption, and candidate acceptance rate.

[0086] Step S63: Verify the reasoning acceleration effect and content consistency based on the reasoning performance indicators.

[0087] It should be noted that the inference acceleration effect refers to the improvement in speed, time consumption, and throughput of this solution compared to traditional autoregressive inference methods. Content consistency verification refers to the process of verifying the completeness, accuracy, and semantic consistency between the accelerated inference output and the original inference output. Inference performance indicators refer to quantitative data such as generation speed, total time consumption, acceptance rate, and number of sequences processed after collection and aggregation. In this embodiment, the system completes two core verifications based on inference performance indicators, ensuring that this solution, while achieving inference acceleration, does not change the model output content, does not reduce generation quality, and does not introduce business deviations, thereby guaranteeing business availability and solution reliability under the constraints of domestic resources. In one possible implementation, the system calculates the speedup ratio by comparing the accelerated indicators with the original inference indicators under the same conditions. In another possible implementation, the system completes consistency verification by comparing the output text before and after acceleration character by character and sequence by sequence.

[0088] Specifically, based on statistically analyzed inference performance metrics, the system performs a horizontal comparison of the speed, time consumption, and throughput of parallel draft accelerated inference with the corresponding metrics of traditional non-accelerated inference under the same conditions, inputs, and lengths. This calculates the acceleration factor and efficiency improvement ratio. Simultaneously, the accelerated final output text is compared with the standard output text without acceleration to verify the consistency of text length, sentence order, core semantics, and key information. Ultimately, a verification conclusion is formed that includes both acceleration effect and content consistency. For example, based on statistically obtained time consumption and generation speed metrics, the system calculates and determines that this solution achieves a significant efficiency improvement compared to traditional inference. The model output before and after acceleration is compared field by field to confirm complete content consistency, no deviation, and no loss, thus completing the overall solution's effectiveness verification.

[0089] This embodiment acquires the inference request to be sent and the corresponding inference parameters, constructs a standardized inference task, and then uses a parallel draft model and a target model to complete joint inference, realizing the parallel generation and verification of multiple text units. At the same time, it statistically analyzes inference performance indicators and finally verifies the acceleration effect and content consistency based on the indicators. This can effectively alleviate the problem of tight computing resources of domestic graphics cards, reduce inference latency, improve the overall inference efficiency in data synthesis scenarios, and ensure that the generated content is stable and reliable and consistent with the original output.

[0090] In one feasible implementation, the step of verifying the inference acceleration effect and content consistency based on the inference performance indicators includes: Step S71: Calculate the generation efficiency and time consumption indicators during the statistical reasoning process, and calculate the acceleration effect based on the generation efficiency and time consumption indicators; It should be noted that generation efficiency refers to the number of text units that the model can generate per unit of time, time consumption refers to the actual time consumed by the system to process a single or batch inference request, and acceleration effect refers to the improvement in efficiency and time consumption compared to traditional inference methods after adopting the solution of this application. In this embodiment, the system statistically analyzes the generation efficiency and time consumption indicators and calculates the acceleration effect, which can intuitively reflect the degree of optimization of the inference process by the solution of this application with quantitative data, providing an objective basis for the effectiveness of the solution. In one possible implementation, the system performs statistics and comparisons under the same hardware environment, the same input content, and the same generation length. In another possible implementation, the system uses the average of multiple sets of test data to improve the accuracy of the calculation results.

[0091] Specifically, the system collects data in real time throughout the entire reasoning process, including the total number of text units generated, the actual execution time, and the time consumed in a single round of reasoning. Based on this data, it calculates the efficiency of text unit generation per unit time and compares this efficiency with the corresponding indicators of traditional reasoning methods that do not use parallel draft acceleration. Based on the efficiency difference and the time difference, it calculates the acceleration ratio and acceleration multiple, forming a quantifiable acceleration effect.

[0092] For example, system statistics show that the number of text units generated per unit time has significantly increased after acceleration, and the overall time consumption has significantly decreased. Based on this, the calculation shows that the improvement in reasoning efficiency is in line with the expected acceleration effect.

[0093] Step S72: Compare the inference output before and after acceleration to determine the consistency of the content. It should be noted that the inference output before and after acceleration refers to the output text obtained using the parallel draft acceleration scheme of this application, and the standard output text obtained by the traditional inference method without the acceleration scheme. Content consistency refers to the comparison and judgment results between the two sets of outputs in terms of text completeness, information accuracy, semantic coherence, and key content matching. In this embodiment, the system compares the output content before and after acceleration to ensure that this application improves inference speed without changing the original output quality and business correctness of the model. In one possible implementation, the system uses a precise character-by-character and text-by-text comparison; in another possible implementation, the system combines semantic verification to assist in confirming content consistency.

[0094] Specifically, the system performs a full field comparison between the accelerated inference output text and the standard output text obtained through traditional methods. It checks whether the text length, sentence order, core semantics, key information, and response content are completely identical, without missing parts, modifications, deviations, or distortions. Based on the comparison results, it determines whether the content is identical or differs. For example, the system compares the inference output before and after acceleration, sentence by sentence, confirming that the two texts are completely identical with no differences, thus obtaining a result indicating content consistency.

[0095] Step S73: Complete the overall solution verification based on the acceleration effect and the content consistency results.

[0096] It should be noted that the overall solution verification refers to the process of comprehensively confirming the feasibility, effectiveness, stability, and practicality of the model inference scheme of this application by integrating the two core indicators of acceleration effect and content consistency. In this embodiment, the system combines acceleration effect and content consistency to complete the verification, ensuring that the scheme simultaneously meets the two core requirements of efficiency improvement and reliable output, and can be stably deployed and used in a domestically produced hardware environment. In one possible implementation, the system uses both achieving the acceleration effect target and complete content consistency as verification pass conditions. In another possible implementation, the system combines the results of multiple rounds of batch testing to form a comprehensive verification conclusion.

[0097] Specifically, the system integrates and judges the determined acceleration effect and content consistency results to confirm that the solution can improve inference efficiency while maintaining the same output content, meeting the operational requirements of data synthesis scenarios under the resource constraints of domestic graphics cards. This completes the validity and usability verification of the overall solution. For example, if the system confirms that the acceleration effect meets the standard and the content consistency is qualified, it determines that the solution meets the design goals and business usage requirements, and the overall solution verification is passed.

[0098] In one embodiment, a complete model training and inference acceleration process is constructed for the data intelligent synthesis scenario. The process is divided into three stages: target model training, parallel draft model training, and joint inference acceleration. Each stage is closely connected, with a unified data source and a unified hardware environment, and the complete implementation of parallel draft inference is achieved on the Huawei Ascend graphics card platform.

[0099] During the model training phase, data preparation was first carried out. Based on the data synthesis scenario, a standard question-and-answer pair format of 20,000 original training data was constructed. The original data existed in the form of user questions and assistant responses, used to support subsequent supervised model fine-tuning and parallel draft training. Next, the target model was trained, using Qwen3-14B as the base model. On a quad-card Huawei Ascend 910B NPU hardware environment, supervised fine-tuning training was performed using the LlamaFactory training framework. The training process employed the LoRA parameter fine-tuning method, setting a truncation length adapted to long contexts, an adaptive learning rate, a specified batch size, and a gradient accumulation strategy. The Ascend platform's dedicated pure BF16 acceleration switch was also enabled to adapt to the computing power characteristics of domestically produced hardware. Before training, the original question-and-answer data is formatted and converted into a unified dialogue template structure containing system prompts, user questions, and assistant replies. Then, the text is converted into a sequence of text units by the model's built-in word segmenter to make the data meet the model's input requirements. Finally, the Qwen3-14B-sft fine-tuning target model adapted to the data synthesis task is obtained. This model is used to undertake the final inference output task.

[0100] After training the target model, a parallel draft model was trained using the same set of standardized training data. Qwen3-4B was selected as the base draft model, and the training objective was to enable the model to generate multiple text units in a single forward pass. The data processing flow of the draft model was consistent with that of the target model: the raw data was first converted into a unified dialogue template format before entering the word segmentation stage. Unlike the single-step prediction of the target model, the parallel draft model treats multiple consecutive text units as a whole for prediction, achieving multi-step inference capability based on the current context to synchronously infer multiple sets of text units. During training, a dedicated parallel data processing logic was constructed, and a mask placeholder structure was introduced to shield future information, ensuring that the information visibility range was consistent between the training and inference stages and avoiding behavioral deviations. Simultaneously, a conditional text unit discarding strategy was adopted, selectively retaining training text units according to preset rules. This reduced the total amount of training data and computational overhead, improving training efficiency while maintaining the integrity of contextual information and ensuring the effectiveness of the KV cache structure. After the above multi-step parallel training and optimization, a Qwen3-4B-draft parallel draft model with the ability to synchronously generate multiple text units was obtained, used to quickly provide candidate content during the inference stage.

[0101] During the inference phase, the vllm-ascend inference framework, adapted for the Huawei Ascend environment, is used to construct a joint inference architecture with the trained target model and the parallel draft model, thereby accelerating inference. During inference, the target model generates text units one by one in an autoregressive manner, while the parallel draft model generates multiple candidate text units simultaneously, which serve as the validation candidate set for the target model. The target model performs parallel verification, probability judgment, and rationality screening on the multiple candidate text units output by the draft model. Qualified candidate text units are directly adopted, and the subsequent inference process continues based on the adoption results, reducing the number of independent forward inference steps by the target model, thus reducing overall inference time and improving inference throughput. The inference process supports configuration of parallel candidate text unit quantity, tensor parallel dimension, generation length, sampling parameters, and other configuration items, which can be flexibly adjusted according to hardware resources and business scenarios. This effectively improves inference efficiency in data synthesis scenarios while ensuring consistency of output content.

[0102] This embodiment calculates the acceleration effect by statistically analyzing generation efficiency and time consumption indicators, compares the output content before and after acceleration to determine the consistency results, and then combines the two indicators to complete the overall solution verification. It can objectively prove that the solution of this application can improve the inference speed while ensuring the stability and accuracy of the output content, and effectively adapt to the hardware environment and resource-constrained scenarios of domestic graphics cards.

[0103] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0104] This application also provides a model inference apparatus, the model inference apparatus comprising: The acquisition module is used to acquire the regularized training data adapted to the data synthesis scenario, as well as the target model and draft model to be deployed. The training module is used to train the draft model in parallel based on the regularized training data and the preset parallel decoding strategy, so as to obtain a parallel draft model that can generate multiple candidate text units at one time. The adaptation module is used to modify the source code of the inference framework adapted to domestic graphics cards, insert parallel draft decoding logic, and obtain a customized inference framework that supports parallel draft inference. The inference module is used to perform joint inference between the target model and the parallel draft model based on the customized inference framework, accelerate the processing of the inference request to be sent, and complete the effect verification. The model inference apparatus provided in this application, employing the model inference method in the above embodiments, can solve the technical problems in the background art. Compared with the prior art, the beneficial effects of the model inference apparatus provided in this application are the same as those of the model inference method provided in the above embodiments, and other technical features in the model inference apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0105] This application provides a model inference device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the model inference method in the above embodiment 1.

[0106] The following is for reference. Figure 4 The diagram illustrates a structural schematic suitable for implementing the model inference device of the embodiments of this application. The model inference device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4The model inference device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0107] like Figure 4 As shown, the model inference device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the model inference device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the model inference device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows model inference devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0108] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0109] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0110] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the model inference methods provided by the above methods.

[0111] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0113] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A model reasoning method, characterized in that, include: Acquire well-organized training data suitable for the data synthesis scenario, as well as the target model and draft model to be deployed; Based on the regularized training data and the preset parallel decoding strategy, the draft model is trained in parallel to obtain a parallel draft model that can generate multiple candidate text units at once. The source code of the inference framework adapted to domestic graphics cards was modified, and parallel draft decoding logic was inserted to obtain a customized inference framework that supports parallel draft inference. Based on the customized inference framework, the target model and the parallel draft model are jointly inferred to accelerate the inference request to be sent and to verify the effect.

2. The model reasoning method as described in claim 1, characterized in that, The acquisition of the standardized training data for the adapted data synthesis scenario, as well as the target model and draft model to be deployed, includes: Obtain the original question-and-answer text data in the data synthesis scenario described above; The original question-and-answer text data is cleaned and format-converted to obtain well-organized training data suitable for model training. Based on the regularized training data, a target model for executing the final output and a draft model for generating candidate content are determined.

3. The model reasoning method as described in claim 1, characterized in that, The step of training the draft model in parallel based on the regularized training data and a preset parallel decoding strategy to obtain a parallel draft model capable of generating multiple candidate text units at once includes: Based on the regularized training data, a multi-step parallel prediction task is constructed; Based on the preset parallel decoding strategy, a mask placeholder structure and a conditional discarding mechanism are introduced; The draft model is trained based on the mask placeholder structure and the conditional discarding mechanism to obtain a parallel draft model that can generate multiple candidate text units at once.

4. The model reasoning method as described in claim 3, characterized in that, The training of the draft model based on the mask placeholder structure and the conditional discarding mechanism includes: The training data is divided into independent parallel subtasks based on the regularized training data; The mask placeholder structure is used to shield future information during the training process in order to maintain consistency between training and inference. The conditional discarding mechanism reduces training overhead and maintains the integrity of contextual information, enabling iterative parallel training of the draft model.

5. The model reasoning method as described in claim 1, characterized in that, The source code of the inference framework adapted to domestic graphics cards is modified, and parallel draft decoding logic is inserted to obtain a customized inference framework that supports parallel draft inference, including: Obtain the source code of the native inference framework adapted to the domestic graphics card environment; Add decoding call logic for parallel draft inference tasks to the decoding execution flow of the original inference framework; Based on the added decoding call logic, compatibility adaptation with domestic hardware environments was completed, resulting in a customized inference framework that supports parallel draft inference.

6. The model reasoning method as described in claim 1, characterized in that, The process, based on the customized inference framework, involves jointly inferring the target model and the parallel draft model to accelerate the inference request to be sent and verify the effect, including: Obtain the inference request to be sent and the inference parameters corresponding to the inference request; Multiple candidate text units are generated through the parallel draft model, and the target model performs parallel verification and adoption to complete the inference output and statistical inference performance indicators based on the verification and adoption results. Based on the aforementioned inference performance metrics, the inference acceleration effect and content consistency verification were completed.

7. The model reasoning method as described in claim 6, characterized in that, The verification of inference acceleration effect and content consistency based on the inference performance indicators includes: The generation efficiency and time consumption indicators in the statistical reasoning process are analyzed, and the acceleration effect is calculated based on the generation efficiency and time consumption indicators. Compare the inference output before and after acceleration to determine the consistency of the content; The overall solution was verified based on the acceleration effect and the consistency of the content.

8. A model reasoning device, characterized in that, include: The acquisition module is used to acquire the regularized training data adapted to the data synthesis scenario, as well as the target model and draft model to be deployed. The training module is used to train the draft model in parallel based on the regularized training data and the preset parallel decoding strategy, so as to obtain a parallel draft model that can generate multiple candidate text units at one time. The adaptation module is used to modify the source code of the inference framework adapted to domestic graphics cards, insert parallel draft decoding logic, and obtain a customized inference framework that supports parallel draft inference. The inference module is used to perform joint inference between the target model and the parallel draft model based on the customized inference framework, accelerate the inference request to be sent, and complete the effect verification.

9. A model reasoning device, characterized in that, The model inference device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the model inference method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the model reasoning method as described in any one of claims 1 to 7.

Images