Data processing methods, electronic devices, storage media and chip systems

Abstract

This application discloses a data processing method, an electronic device, a storage medium, and a chip system. The method is applied to an electronic device with a first processor deployed with a target model and a second processor deployed with a draft model. In a multi-task inference scenario, for two tasks to be processed (i.e., a first input text and a second input text), the processing time of the target model running the decoding phase of the first input text (i.e., at least one round of the first target decoding operation in N rounds of the first iteration) on the first processor overlaps with the processing time of the draft model running the decoding phase of the second input text (i.e., at least one round of the second draft decoding operation in M ​​rounds of the second iteration). This achieves the goal of parallel execution of the decoding phases of the target model and the draft model, reducing data processing latency, improving data processing efficiency, and thus enhancing the user experience.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a data processing method, electronic device, storage medium, and chip system. Background Technology

[0002] With the continuous development of terminal technology, electronic devices can provide users with a wide variety of services. In practical applications, users input prompt words into the Large Language Model (LLM) within the electronic device. After processing the prompt words, the LLM can display the predicted response content to the user. Speculative sampling is a novel decoding acceleration technique in the field of LLM inference optimization. Unlike traditional autoregressive decoding processes, speculative sampling can predict and generate multiple subsequent tokens for parallel verification, ultimately achieving the effect of generating multiple tokens after one iteration, thereby reducing the number of inference rounds.

[0003] Currently, in multi-task reasoning scenarios, traditional speculative sampling for data processing suffers from high data processing latency and low data processing efficiency, resulting in a poor user experience. Summary of the Invention

[0004] This application provides a data processing method, a data processing device, an electronic device, a storage medium, a chip system, and a computer program product, which can improve and reduce data processing latency, increase data processing efficiency, and enhance user experience.

[0005] In a first aspect, embodiments of this application provide a data processing method applied to an electronic device having a first processor deployed with a target model and a second processor deployed with a draft model. The method includes: performing a first speculative sampling process on an acquired first input text to generate a first output text, wherein the first speculative sampling process includes N rounds of first iterations, the i-th round of first iterations including running the draft model to generate an i-th round of first draft decoding operation based on the i-th round of first draft data, and running the target model to verify the i-th round of candidate words to generate an i-th round of first iteration results; when i equals 1, the i-th round of first draft data includes a first word sequence corresponding to the first input text; when i is an integer greater than 1 and less than or equal to N, the i-th round of first draft data includes a first word sequence and the (i-1)-th round of first iteration results corresponding to the previous (i-1) rounds of first iterations, and the first output text is obtained by concatenating the N-th round of first iteration results corresponding to the N-th round of first iterations, where N is a positive integer; performing a second speculative sampling process on an acquired second input text to generate a first output text. The second output text is generated, wherein the second speculative sampling process includes M rounds of second iterations, the j-th round of second iterations includes the j-th round of second draft decoding operation of running the draft model to generate j-th round candidate words based on the j-th round of second draft data, and the j-th round of second target decoding operation of running the target model to verify the j-th round candidate words and generate the j-th round of second iteration results; when j equals 1, the j-th round of second draft data includes the second word sequence corresponding to the second input text; when j is an integer greater than 1 and less than or equal to M, the j-th round of second draft data includes the second word sequence and the (j-1) rounds of second iteration results corresponding to the previous (j-1) rounds of second iterations, and the second output text is obtained by concatenating the M-th round of second iteration results corresponding to the M-th round of second iterations, where M is a positive integer; wherein the processing time of the N rounds of first iterations and the processing time of the M rounds of second iterations have at least one first overlap time, the at least one first overlap time includes the overlap time between the processing time of at least one round of first target decoding operation in the N rounds of first iterations and the processing time of at least one round of second draft decoding operation in the M rounds of second iterations.

[0006] There is at least one first overlap between the processing time of the first iteration in N rounds and the processing time of the second iteration in M ​​rounds. The first first overlap includes the overlap between the processing time of at least one round of the first target decoding operation in the first iteration in N rounds and the processing time of at least one round of the second draft decoding operation in the second iteration in M ​​rounds. This means that at least one round of the first target decoding operation in the first iteration in N rounds and at least one round of the second draft decoding operation in the second iteration in M ​​rounds are executed in parallel.

[0007] A scenario where an electronic device performs speculative sampling on multiple (e.g., first input text and second input text) input texts can be considered a reasoning scenario where the electronic device performs multiple tasks.

[0008] At least one first overlap time includes the overlap time between the processing time of at least one round of the first target decoding operation in N rounds of the first iteration and the processing time of at least one round of the second draft decoding operation in M ​​rounds of the second iteration. The duration of at least one first overlap time is not limited. In practical applications, the calculation process of the draft model is relatively simple and the calculation time is short; the calculation process of the target model is more complex and the calculation time is longer. When the processing time of at least one round of the first target decoding operation is greater than the processing time of at least one round of the second draft decoding operation, the overlap time between the processing time of at least one round of the first target decoding operation in N rounds of the first iteration and the processing time of at least one round of the second draft decoding operation in M ​​rounds of the second iteration can be the processing time of at least one round of the second draft decoding operation.

[0009] In the technical solution provided in this application, in a multi-task reasoning scenario, the target model is deployed on one processor (i.e., the first processor) of an electronic device, and the draft model is deployed on another processor (i.e., the second processor) of the electronic device, thereby achieving the decoupling of the target model and the draft model. Based on this, during data processing, for the two tasks to be processed (i.e., the first input text and the second input text), by setting the processing time of the target model on one processor running the decoding phase of one input text (i.e., the first input text) (i.e., at least one round of the first target decoding operation in N rounds of the first iteration) to overlap with the processing time of the draft model on the other processor running the decoding phase of another input text (i.e., the second input text) (i.e., at least one round of the second draft decoding operation in M ​​rounds of the second iteration) (i.e., at least one first overlap time), the goal of parallel execution of the decoding phase of the target model and the decoding phase of the draft model in a multi-task reasoning scenario is achieved. Compared with the highly serial execution process of traditional speculative sampling (i.e., both the target large model and the draft model follow a serial inference paradigm), the above-mentioned technical solution provided in this application can save computational overhead (i.e., the computational overhead of the aforementioned overlapping time), reduce data processing latency, and accelerate model inference, thereby improving data processing efficiency and enhancing user experience.

[0010] In one possible implementation, at least one first target decoding operation in the N rounds of the first iteration is the i-th round of the first target decoding operation, and at least one second draft decoding operation in the M rounds of the second iteration is the i-th round of the second draft decoding operation; under the condition of satisfying the first condition, at least one first overlap time also includes the processing time of the (i+1)-th round of the first draft decoding operation and the overlap time with the processing time of the i-th round of the second target decoding operation; wherein, satisfying the first condition includes: i is a positive integer less than or equal to N, M is a positive integer greater than or equal to N, and the N rounds of the first iteration include the (i+1)-th round of the first draft decoding operation; or, satisfying the first condition includes: i is a positive integer less than or equal to M, M is a positive integer less than N, and N is a positive integer greater than 1.

[0011] The first iteration of round (i+1) is the next iteration after the first iteration of round (i). The first iteration of round (i) includes the first draft decoding operation and the first target decoding operation of round (i). The first iteration of round (i+1) includes the first draft decoding operation and the first target decoding operation of round (i+1).

[0012] The second iteration of round i includes the second draft decoding operation of round i and the second target decoding operation of round i.

[0013] For example, satisfying the first condition includes: i being a positive integer less than or equal to N, M being a positive integer greater than or equal to N, and N rounds of the first iteration including the (i+1)th round of the first draft decoding operation. For instance, at least one first overlap time in the above example may include... Figure 9 The three overlapping time periods #1 shown in (b) are not detailed here, but can be found in the relevant description below.

[0014] For example, satisfying the first condition includes: i being a positive integer less than or equal to M, M being a positive integer less than N, and N being a positive integer greater than 1. Exemplarily, at least one first overlap time in the above example may include... Figure 9 The two overlapping time periods #1 shown in (a) are not detailed here, but can be found in the relevant description below.

[0015] In the above-described technical solution provided in this application, in a multi-task reasoning scenario, for two tasks to be processed (i.e., the first input text and the second input text), the processing time of the target model running the decoding phase of one input text (i.e., the first input text) on one processor in the electronic device (i.e., the i-th round of the first target decoding operation) overlaps with the processing time of the draft model running the decoding phase of another input text (i.e., the second input text) on another processor (i.e., the second processor) in the electronic device (i.e., the i-th round of the second draft decoding operation); and the processing time of the target model running the other input text on the first processor overlaps with the processing time of the draft model running the decoding phase of the second input text on the second processor (i.e., the second processor) in the electronic device. The processing time of the decoding stage of the second input text (i.e., the second target decoding operation in the i-th round) overlaps with the processing time of the decoding stage of the first input text (i.e., the first draft decoding operation in the (i+1)-th round) of the draft model on another processor (i.e., the second processor) in the electronic device. This achieves the goal of highly parallel execution of the decoding stage of the target model and the decoding stage of the draft model in a multi-task reasoning scenario. It can further save computational overhead (i.e., the computational overhead of the two overlapping times mentioned above), reduce data processing latency, accelerate model reasoning, thereby further improving data processing efficiency and enhancing the user experience.

[0016] In another possible implementation, the first speculative sampling process further includes a first target pre-filling process and a first draft pre-filling process performed before the first iteration in N rounds. The first lexical sequence includes Q first target lexical blocks, where Q is a positive integer greater than 1. The first target pre-filling process includes running a target model to generate a k-th first target block result based on the k-th first target lexical block, thereby generating the first target pre-filling result; where k is a positive integer less than or equal to Q. The first draft pre-filling process includes running a draft model to generate a k-th first draft lexical block based on the k-th first draft lexical block. The draft is segmented into blocks to generate the first draft pre-filled result; the kth first draft word block is determined based on the corresponding kth first target word block; the i-th round of first draft data includes the first word sequence, the first target pre-filled result, and the first draft pre-filled result; among them, the processing time of the target model generating the wth first target segmented result based on the wth first target word block overlaps with the processing time of the draft model generating the (w-1)th first draft segmented result based on the (w-1)th first draft word block, where w is a positive integer greater than 1 and less than or equal to Q.

[0017] The pre-filling process for the target model corresponding to each input text includes multiple (i.e., Q) block target pre-filling processes, and the pre-filling process for the draft model corresponding to each input text includes multiple (i.e., Q) block draft pre-filling processes. The first target pre-filling result includes Q first target block results corresponding to Q first target word blocks. The first draft pre-filling result includes Q first draft block results corresponding to Q first draft word blocks.

[0018] Optionally, the first target pre-filling process further includes a first data preprocessing to generate a first word sequence based on the first input text, wherein the processing time of the first data preprocessing is earlier than the processing time of generating the kth first target block result.

[0019] In the technical solution provided in this application, by dividing the first word sequence corresponding to the first input text into multiple word blocks, and setting the processing time of the target model performing pre-filling processing on the word block corresponding to the first input text (i.e., the w-th first target word block) to overlap with the processing time of the draft model performing draft pre-filling processing on the draft word block corresponding to the word block (i.e., the (w-1)-th first draft word block), the goal of parallel execution of the pre-filling stage of the target model and the pre-filling stage of the draft model is achieved. This not only further saves computational overhead (i.e., the computational overhead of the aforementioned overlapping time), but also reduces the waiting time during the first round of decoding of the target model (i.e., the first round of decoding of the first target operation). In summary, based on the above technical solution, data processing latency can be further reduced, model inference can be accelerated, thereby further improving data processing efficiency and enhancing the user experience.

[0020] In another possible implementation, the second speculative sampling process further includes a second target pre-filling process and a second draft pre-filling process performed before the Nth round of the second iteration. The second lexical sequence includes Q2 second target lexical blocks, where Q2 is a positive integer greater than 1. The second target pre-filling process includes running the target model to generate a k2-th second target block result based on the k2-th second target lexical block, thereby generating the second target pre-filling result; where k2 is a positive integer less than or equal to Q2. The second draft pre-filling process includes running the draft model to generate a k2-th second draft lexical block based on the k2-th second draft lexical block. The draft is segmented into blocks to generate the second draft pre-filled result; the k2th second draft word block is determined based on the corresponding k2th second target word block; the i-th round of second draft data includes the second word sequence, the second target pre-filled result, and the second draft pre-filled result; among them, the processing time of the target model generating the w2th second target segmented result based on the w2th second target word block overlaps with the processing time of the draft model generating the (w2-1)th second draft segmented result based on the (w2-1)th second draft word block, where w2 is a positive integer greater than 1 and less than or equal to Q2.

[0021] Optionally, the second target pre-filling process also includes a second data preprocessing that generates a second word sequence based on the second input text, wherein the processing time of the second data preprocessing is earlier than the processing time for generating the k2th second target block result.

[0022] In the technical solution provided in this application, by dividing the second word sequence corresponding to the second input text into multiple word blocks, and setting the processing time of the target model performing pre-filling processing on the word block corresponding to the second input text (i.e., the w2th second target word block) to overlap with the processing time of the draft model performing draft pre-filling processing on the draft word block corresponding to the word block (i.e., the (w2-1th)th second draft word block), the goal of parallel execution of the pre-filling stage of the target model and the pre-filling stage of the draft model is achieved. This not only further saves computational overhead (i.e., the computational overhead of the aforementioned overlapping time), but also reduces the waiting time during the first round of decoding of the target model (i.e., the first round of second target operation decoding). In summary, based on the above technical solution, data processing latency can be further reduced, model inference accelerated, thereby further improving data processing efficiency and enhancing the user experience.

[0023] In another possible implementation, the method further includes: performing a third speculative sampling process on the acquired third input text to generate a third output text, wherein the third speculative sampling process includes L rounds of third iterations, the r-th round of third iterations includes running a draft model to generate r-th round candidate lexical units based on the r-th round of third draft data, and running a target model to verify the r-th round candidate lexical units to generate the r-th round of third iteration results, and running a target model to verify the r-th round candidate lexical units and generate the r-th round of third iteration results; when r equals 1, the r-th round of third draft data includes the third lexical unit sequence corresponding to the third input text; when r is an integer greater than 1 and less than or equal to L, the r-th round of third draft data includes the third lexical unit sequence and the (r-1)-th round of third iteration results corresponding to the previous (r-1) rounds of third iterations, and the third output text is a result of the L-th round of third iterations corresponding to the L-th round of third iterations. The result after line concatenation is L, where L is a positive integer. Specifically, after the completion of the M rounds of second iterations, the completion of the first H rounds of first iterations in the N rounds of first iterations, and the remaining (NH) rounds of first iterations excluding the first H rounds of first iterations not being executed, the processing time of the L rounds of third iterations overlaps with the processing time of the remaining (NH) rounds of first iterations by at least one second overlap. This second overlap includes the overlap between the processing time of at least one round of first target decoding operations in the remaining (NH) rounds of first iterations and the processing time of at least one round of third draft decoding operations. The first overlap includes the overlap between the processing time of at least one round of first target decoding operations in the first H rounds of first iterations and the processing time of at least one round of second draft decoding operations in the M rounds of second iterations, where H is a positive integer less than N.

[0024] In the technical solution provided in this application, when all decoding operations corresponding to the second input text (i.e., the M rounds of the second iteration process) have been completed, some decoding operations corresponding to the first input text (i.e., the first H rounds of the first iteration process) have been completed, and the remaining decoding operations corresponding to the first input text (i.e., the remaining (NH) rounds of the first iteration) have not yet been executed, and the electronic device needs to perform speculative sampling on the new third input text, the target model located in the first processor executes at least one round of the first target decoding operation in the remaining (NH) rounds of the first iteration corresponding to the first input text, while the draft model located in the second processor executes at least one round of the third draft decoding operation corresponding to the third input text. This achieves dynamic task filling and avoids the phenomenon of the electronic device serially executing a single task (e.g., the remaining (NH) rounds of the first iteration process corresponding to the first input text). In addition, in the technical solution provided in this application, the parallel execution of the decoding stage of the target model and the decoding stage of the draft model can further save computational overhead (i.e., the computational overhead of the aforementioned overlapping time), reduce data processing latency, accelerate model inference, thereby improving data processing efficiency and enhancing the user experience.

[0025] In another possible implementation, the third speculative sampling process further includes a third target pre-filling process and a third draft pre-filling process performed before the third iteration of L rounds. The third lexical sequence includes P third target lexical blocks, where P is a positive integer greater than 1. The third target pre-filling process includes running the target model to generate the h-th third target pre-filling result based on the h-th third target lexical block, thereby generating the third target pre-filling result; where h is a positive integer less than or equal to P. The third draft pre-filling process includes running the draft model based on the h-th third target lexical block. The draft lexical block generates the h-th third draft pre-fill result to generate the third draft pre-fill result; the h-th third draft lexical block is determined based on the corresponding h-th third target lexical block; the j-th round of third draft data includes the third lexical sequence, the third target pre-fill result, and the third draft pre-fill result; among them, the processing time for running the target model to generate the v-th third target pre-fill result overlaps with the processing time for running the draft model to generate the (v-1)-th third draft pre-fill result, where v is a positive integer greater than 1 and less than or equal to P.

[0026] In the technical solution provided in this application, by dividing the third word sequence corresponding to the third input text into multiple word blocks, and setting the processing time of the target model performing pre-filling processing on the word block corresponding to the third input text (i.e., the v-th third target word block) to overlap with the processing time of the draft model performing draft pre-filling processing on the draft word block corresponding to the word block (i.e., the (v-1)-th third draft word block), the goal of parallel execution of the pre-filling stage of the target model and the pre-filling stage of the draft model is achieved. This not only further saves computational overhead (i.e., the computational overhead of the aforementioned overlapping time), but also reduces the waiting time during the first round of decoding of the target model (i.e., the first round of third target operation decoding). In summary, based on the above technical solution, data processing latency can be further reduced, model inference accelerated, and thus data processing efficiency further improved, further enhancing the user experience.

[0027] In another possible implementation, the first processor is a neural network processor and the second processor is a graphics processor.

[0028] In the above-mentioned technical solution provided in this application, the neural network processor as the first processor is more compatible with the target model with a more complex calculation process, and the graphics processor as the second processor is more compatible with the draft model with a simpler calculation process. Therefore, based on the above settings, it is beneficial to improve the resource utilization of each processor.

[0029] In another possible implementation, the number of parameters in the draft model is smaller than the number of parameters in the target model.

[0030] Secondly, embodiments of this application provide a data processing apparatus, which includes a unit for performing any of the data processing methods in the first aspect.

[0031] Thirdly, an electronic device is provided, including a unit for performing any of the data processing methods in the first aspect. The electronic device may be a terminal device or a chip within a terminal device. The electronic device may include an input unit and a processing unit.

[0032] When the electronic device is a terminal device, the processing unit may be a processor, and the input unit may be a communication interface; the terminal device may also include a memory for storing computer program code, which, when the processor executes the computer program code stored in the memory, causes the terminal device to perform any of the data processing methods in the first aspect.

[0033] When the electronic device is a chip within a terminal device, the processing unit can be an internal processing unit of the chip, and the input unit can be an output interface, pin, or circuit, etc.; the chip may also include a memory, which can be an internal memory of the chip (e.g., registers, cache, etc.) or an external memory (e.g., read-only memory, random access memory, etc.); the memory is used to store computer program code, and when the processor executes the computer program code stored in the memory, the chip performs any of the data processing methods in the first aspect.

[0034] In one possible implementation, the memory is used to store computer program code; the processor executes the computer program code stored in the memory, and when the computer program code stored in the memory is executed, the processor is used to perform any of the data processing methods in the first aspect.

[0035] Fourthly, a computer-readable storage medium is provided that stores computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the data processing methods described in the first aspect.

[0036] Fifthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform any of the data processing methods described in the first aspect.

[0037] In a sixth aspect, a computer program product is provided, comprising: computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the data processing methods described in the first aspect.

[0038] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0039] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0040] Figure 1 It is a timeline diagram of traditional speculative sampling inference.

[0041] Figure 2 This is a schematic diagram illustrating the application scenario to which the data processing method provided in the embodiments of this application is applicable.

[0042] Figure 3 This is a schematic diagram of the user interface of an electronic device to which the data processing method provided in this application is applicable.

[0043] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0044] Figure 5 This is a schematic diagram of a software system for an electronic device provided in an embodiment of this application.

[0045] Figure 6 This is a timing diagram of the decoding stage in a speculative sampling inference provided in an embodiment of this application.

[0046] Figure 7 This is a schematic diagram of a data processing method 700 provided in an embodiment of this application.

[0047] Figure 8 This is a timing diagram of the decoding stage in a speculative sampling inference provided in an embodiment of this application.

[0048] Figure 9 This is a timing diagram of the decoding stage in a speculative sampling inference provided in an embodiment of this application.

[0049] Figure 10 This is a schematic diagram of a data processing method 1000 provided in an embodiment of this application.

[0050] Figure 11This is a timing diagram of the pre-filling stage in speculative sampling inference provided in an embodiment of this application.

[0051] Figure 12 This is a schematic diagram of a data processing method 1200 provided in an embodiment of this application.

[0052] Figure 13 This is a timing diagram of the pre-filling stage in speculative sampling inference provided in an embodiment of this application.

[0053] Figure 14 This is a timing diagram of a speculative sampling inference provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] In this application, the target model is also referred to as the original model, original target model, target large model, target validation model, validation model, autoregressive model, autoregressive large model, or autoregressive decoding model. The draft model is also referred to as the speculative model, target draft model, speculative small model, or approximate model. The prefill stage is also referred to as the first character or encoding stage. The decoding stage is also referred to as the generation stage. Speculative sampling is also referred to as speculative decoding or predictive decoding, which will not be emphasized further below.

[0056] To facilitate a better understanding of the technical solutions provided in this application, the relevant terms used in this application will be introduced first.

[0057] 1. Prompt

[0058] Prompt words, also known as input text, user input information, or prompt text, refer to the question text that the user inputs into the model. Prompt words are used to request the execution of a user task, which can be, but is not limited to, text generation or image recognition tasks. For example, a prompt word could be "What's the weather like today?" Or, "Please identify the text information in a given image from the image library."

[0059] 2. Token

[0060] In the field of natural language processing, a token refers to a lexical unit, which is the smallest semantic unit in text, and can be a word, phrase, or character. Large language models typically convert input sequences (such as images or text) into token sequences for processing and output generation. Tokens can also be called tokens, etc., without limitation.

[0061] 3. End of Sequence (EOS)

[0062] EOS indicates that the generated output text sequence has reached its termination point, and the model stops generating new tokens after encountering an EOS token.

[0063] 4. Key-Value Cache (KV Cache)

[0064] Key-value (KV) caching is a common optimization technique used in the inference process of large models (especially transformer models), primarily to accelerate computational efficiency in autoregressive generation tasks (such as text generation and image recognition). The core idea of ​​KV caching is to cache previously calculated keys and values, calculate only the keys and values ​​of newly added tokens, and then reuse the cached results, thereby avoiding redundant calculations and significantly reducing the computational load and latency of inference.

[0065] 5. The reasoning process of Large Language Models (LLM)

[0066] LLM (Language Modeling) has demonstrated powerful modeling capabilities in tasks such as natural language processing, intelligent dialogue, and semantic understanding, and has been widely deployed in various intelligent application systems. The inference process of LLM includes a prefill phase and a decoding phase. In the prefill phase of LLM, after the model computes the prompt words (e.g., through forward pass), it generates the first token in the output text, and caches the key-value data (KV) of the token corresponding to the prompt word in a KVCache for use in the subsequent decoding phase. The prefill phase is responsible for processing the complete content of the prompt words and is computationally intensive; therefore, it is a computationally intensive phase. In the decoding phase of LLM, the model uses an autoregressive generation mechanism based on the token generated in the previous step and the KV data cached in the KV Cache, that is, it generates subsequent tokens sequentially according to time order (i.e., tokens following the first token in the output text). The above-mentioned decode phase generates subsequent tokens one by one through an autoregressive method. Although the amount of computation per step is small, the generation of each new token must repeatedly access the KV Cache corresponding to all previously generated tokens. Therefore, the decode phase is a memory-bound phase (also known as a memory access-intensive phase).

[0067] The inference phase of the LLM mentioned above adopts an autoregressive generation mechanism, which generates output tokens one by one in chronological order. This makes the model inference process highly serialized and computationally dependent, resulting in high overall latency, low computational efficiency, and high consumption of computational resources (failing to fully utilize the processor's computing power). Consequently, the inference speed is slow and it is difficult to meet the application scenarios with strict requirements for interactive real-time performance and low power consumption.

[0068] 6. Speculative sampling

[0069] Speculative sampling, as an inference acceleration technique, can address some of the problems associated with LLM inference. The core idea of ​​speculative sampling inference is as follows: First, a small draft model (DM) continuously generates multiple candidate tokens based on an autoregressive generation mechanism (i.e., an autoregressive serial sampling process). Then, a large target model (TM) performs parallel verification (evaluation) of these candidate tokens. If the candidate tokens pass verification, the next iteration continues; if they fail verification, the target model regenerates a token and the next iteration continues, ultimately achieving the effect of generating multiple tokens after one iteration of the target model. Compared to schemes without speculative sampling, the speculative sampling scheme significantly reduces the number of autoregressive calculations per token in the target model (i.e., reduces the number of iterations in the target model), greatly reducing inference latency, thereby improving throughput and accelerating the model's inference process.

[0070] The speculative sampling reasoning process includes the prefill phase (abbreviated as prefill phase) and the decode phase (abbreviated as decode phase).

[0071] The prefill stage will be introduced below.

[0072] The prefill phase includes the prefill phase of the target model (abbreviated as prefill(TM)) and the prefill phase of the draft model (abbreviated as prefill(DM)), such as Figure 1 The prefill phase shown is executed after the prefill phase of the target model; that is, the two prefill phases are executed sequentially.

[0073] The prefill phase of the target model is used to compute the initial target lexical sequence corresponding to the prompt word, generate the first target token (i.e., the first token in the final output text), and the contextual information of the initial target lexical sequence (e.g., the token corresponding to the initial target lexical sequence). , The matrix represents the initial target word sequence, where the initial target word sequence is the sequence obtained by converting each word in the prompt into a word unit. The contextual information of the initial target word sequence is used in the subsequent decoding stage of the target model. Optionally, the prefill stage of the target model is also used to convert the prompt into the initial target word sequence.

[0074] The prefill phase of the draft model is used to compute the initial draft lexical sequence, generate the first candidate token, and the contextual information of the initial draft lexical sequence (e.g., the token corresponding to the initial draft lexical sequence). , (a matrix), where the contextual information of the initial draft lexical sequence is used in the decoding stage of the subsequent draft model.

[0075] The initial draft lexical sequence is a lexical sequence determined based on the initial target lexical sequence and the first target token. The initial draft lexical sequence is described below.

[0076] In implementation method one, since the initial target word sequence has a small sequence length, it is not necessary to perform block processing on the initial target word sequence. For example, the initial draft word sequence can be a word sequence obtained by appending the first target token to the initial target word sequence. For example, if the initial target word sequence is [1 2 3] and the first target token is [4], the initial draft word sequence is [1 2 3 4]. For example, the initial draft word sequence can be a word sequence obtained by appending the first target token to the initial target word sequence 0, where the initial target word sequence 0 is the sequence obtained by deleting the first word in the initial target word sequence. For example, if the initial target word sequence is [1 2 3] and the first target token is [4], the initial draft word sequence is [2 3 4].

[0077] In implementation method two, the initial target word sequence is relatively long. Therefore, the initial target word sequence can be divided into smaller chunks (Q), where Q is a positive integer greater than or equal to 2. Figure 1The initial target word sequence is shown, corresponding to two target word blocks T0 and T1. In one example, the initial target word sequence includes Q target word blocks, and the initial draft word sequence includes Q draft word blocks that correspond one-to-one with the Q target word blocks. The k-th draft word block is determined based on the k-th target word block and the first token. The k-th draft word block is obtained by continuating the first token after the k-th target word block 0. The k-th target word block 0 can be obtained by deleting the token at the first position in the k-th target word block. When k is an integer greater than or equal to 1 and less than Q, the first token is the first token in the (k+1)-th target word block; when k equals M, the first token is the first target token, k=1,2,...,Q. In another example, the initial target word sequence includes Q target word blocks, and the initial draft word sequence includes Q draft word blocks that correspond one-to-one with the Q target word blocks. The k-th draft word block is the k-th target word block, where k = 1, 2, ..., Q.

[0078] In the second implementation described above, the prefill stage of the target model is used to calculate the initial target lexical sequence, generating the context information of the initial target lexical sequence and the first target token. For example, it may include the following steps: the target model calculates the k-th target lexical block to obtain its context information; the target model concatenates the context information of the Q target lexical blocks according to the time order in which they were generated, obtaining the context information of the initial target lexical sequence; the target model then processes the context information of the Q target lexical blocks and the last token in the Q target lexical blocks... Attention is calculated using a matrix to obtain the first target token. Similarly, the prefill stage of the draft model is used to calculate the initial draft lexical sequence (i.e., the initial sequence input to the draft model) to generate the context information of the initial draft lexical sequence and the first candidate token. For example, this can include the following steps: the draft model calculates the k-th draft lexical block to obtain its context information; the draft model concatenates the context information of the Q draft lexical blocks according to the time order in which they were generated to obtain the context information of the initial draft lexical sequence; the draft model then processes the context information of the Q draft lexical blocks and the last token in the Q draft lexical blocks... Attention is calculated on the matrix to obtain the first candidate token. It should be understood that the above prefill stage is also known as the chunk prefill stage, which decomposes a long sequence prefilling task (such as the initial target word sequence) into multiple smaller "chunks" (such as Q target word chunks) to improve processing efficiency and resource utilization.

[0079] For example, taking the initial target word sequence [1 2 3 4 5 6 7 8 9 10] in the example above, and the preset block size is 5 tokens, with the first target token being

[11] , after processing the initial target word sequence [1 2 3 4 5 6 7 8 9 10] into blocks, the two target word blocks obtained are target word block T0 [1 2 3 4 5] and target word block T1 [6 7 8 9 10]; and the two draft word blocks obtained are draft word block C0 [2 3 4 5 6] and draft word block C1 [7 8 9 10 11].

[0080] The following section introduces the decoding stage.

[0081] The decode phase consists of n iterations. The i-th iteration includes the i-th decode phase of the draft model (abbreviated as guess(DM)_i) and the i-th decode phase of the target model (abbreviated as decode(TM)_i). The i-th iteration can be any one of the n iterations, such as... Figure 1 The decode phase is shown. The i-th decode phase of the draft model uses an autoregressive generation mechanism to generate the i-th round candidate tokens based on the i-th draft lexical sequence. When i equals 1, the i-th draft lexical sequence can be the sequence obtained by appending the first candidate token to the initial draft lexical sequence; when i is an integer greater than or equal to 2, the i-th draft lexical sequence can be the sequence obtained by appending the result of the (i-1)-th iteration to the (i-1)-th draft lexical sequence. The i-th decode phase of the target model is used to perform parallel verification of the i-th round candidate tokens to obtain the i-th round iteration result.

[0082] The relevant technical solutions provided in this application are described below with reference to the accompanying drawings.

[0083] Currently, in multi-task reasoning scenarios, traditional speculative sampling for data processing suffers from high data processing latency and low data processing efficiency, resulting in a poor user experience.

[0084] The following example illustrates traditional speculative sampling in a multi-task inference scenario, where two tasks are executed corresponding to two input texts, with each input text corresponding to one task. Specifically, in traditional speculative sampling, when speculative sampling is performed on two input texts, the electronic device first performs speculative sampling on one input text to obtain its corresponding input text, and then performs speculative sampling on the other input text to obtain its corresponding output text. The process of speculative decoding for each input text can be found in [link to relevant documentation]. Figure 1 The timing diagram shown illustrates that the electronic device first runs the target model deployed on the processor to perform the target pre-filling process corresponding to the input text (i.e., Figure 1 The prefill(TM) process in the processor is then executed. Following this, the draft model deployed on the processor continues to run the draft prefill process corresponding to the input text (i.e., the prefill(TM) process). Figure 1 The prefill (DM) process in the input text is now complete. It's evident that the prefill processes for the target model and the draft model are executed in a highly sequential manner (i.e., the target model and the draft model follow a sequential inference paradigm). Afterwards, the process corresponding to the input text is executed, including at least one iteration (e.g., ...). Figure 1 The decoding process (shown as n iterations), each iteration (e.g.) Figure 1 The first iteration shown includes the draft model on the processor predicting a batch of candidate tokens based on the already generated tokens during the draft model decoding process (e.g., ...). Figure 1 The draft model shown is the first iteration guess(DM)_1 process, and the target model decoding process running on the processor to verify the batch of candidate tokens and generate the results of this iteration (e.g. Figure 1 The first iteration of the target model (decode(TM)_1) is shown, indicating that the decoding process of the target model and the draft model are also executed in a highly serial manner (i.e., the target model and the draft model follow a serial inference paradigm). Therefore, it is evident that the aforementioned traditional speculative sampling suffers from high data processing latency and low data processing efficiency, resulting in a poor user experience.

[0085] Therefore, to solve the above problems, this application provides a data processing method, a data processing device, an electronic device, a storage medium, a chip system, and a computer program product. The technical solution provided by this application includes: on the one hand, in a multi-task reasoning scenario, deploying the target model in one processor (i.e., the first processor) of an electronic device, and deploying the draft model in another processor (i.e., the second processor) of the electronic device, thereby achieving the purpose of decoupling the target model and the draft model. Based on this, during data processing, for the two tasks to be processed (i.e., the first input text and the second input text), by setting the processing time of the target model running the decoding phase of one input text (i.e., the first input text) on one processor in the electronic device (i.e., at least one round of the first target decoding operation in N rounds of the first iteration) to overlap with the processing time of the draft model running the decoding phase of another input text (i.e., the second input text) on another processor (i.e., the second processor) on the same electronic device (i.e., at least one round of the second draft decoding operation in M ​​rounds of the second iteration) (i.e., at least one first overlap time), the goal of parallel execution of the decoding phase of the target model and the decoding phase of the draft model in a multi-task reasoning scenario is achieved. This can save computational overhead, reduce data processing latency, accelerate model reasoning, thereby improving data processing efficiency and enhancing the user experience. On the other hand, by dividing the word sequence corresponding to the input text (e.g., the first input text, the second input text, and the third input text) into multiple word blocks, and setting the processing time of the target model performing pre-filling processing on the word block corresponding to the input text (e.g., the w-th first target word block) to overlap with the processing time of the draft model performing draft pre-filling processing on the draft word block corresponding to the word block (e.g., the (w-1)-th first draft word block), the goal of parallel execution of the pre-filling stage of the target model and the pre-filling stage of the draft model is achieved. In this way, not only can the computational overhead (i.e., the computational overhead of the aforementioned overlapping time) be further saved, but the waiting time during the first round of decoding of the target model (i.e., the first round of target operation decoding) can also be reduced, thereby further reducing data processing latency, improving data processing efficiency, and further enhancing the user experience.On the other hand, after the task corresponding to the second input text has been completed, and there is still a part of the decoding process (i.e., the remaining (NH) rounds of the first iteration) of the task corresponding to the first input text that has not been executed, when the electronic device needs to execute the task corresponding to the new third input text, by setting the target model located in the first processor to execute at least one round of the first target decoding operation in the remaining (NH) rounds of the first iteration corresponding to the first input text, and the draft model located in the second processor to execute at least one round of the third draft decoding operation corresponding to the third input text, the dynamic task filling is realized, avoiding the phenomenon that the electronic device serially executes a single task (such as the remaining (NH) rounds of the first iteration process corresponding to the first input text), which can further save computational overhead, and at the same time, it is also conducive to improving the resource utilization of the processor.

[0086] The following is combined with Figure 2 and Figure 3 A schematic diagram illustrating the application scenarios to which the data processing method provided in the embodiments of this application is applicable.

[0087] For example, Figure 2 The application scenario shown may include at least one terminal 200A. For example, the number of terminals 200A may be one or more (e.g., 2, 3 or 5, etc.), and there is no limitation thereto.

[0088] In one example, terminal 200A can execute the data processing method provided in this application. A target application is installed and running on terminal 200A, which calls a model (including a target model located in a first processor and a draft model located in a second processor) located on a processor in an electronic device, causing the model to execute the data processing method provided in this application.

[0089] For example, taking terminal 200A as an example Figure 3 Taking the mobile phone shown in (a) as an example, when the mobile phone is in a screen-off charging state, the image library application running on the mobile phone calls the model located on the processor in the electronic device, so that the model processes the data of at least one image in the image library application according to the prompt words and generates the corresponding output text (such as the subject of the object in the image (such as people, landscapes, buildings, etc.), or the user's name and ID number in the image).

[0090] For example, taking terminal 200A as an example Figure 3 Taking the mobile phone shown in Figure (b) as an example, when the mobile phone receives a text message, the text message application running on the mobile phone calls the model located on the processor in the electronic device (including the target model located in the first processor and the draft model located in the second processor), so that the model processes the text message according to the prompt words and generates the corresponding output text (such as schedule information).

[0091] In another example, the above Figure 2 The application scenario shown can also include server 200B, indicated by the dashed line. Server 200B can execute the data processing method provided in this application. Server 200B can perform model inference on the prompt information obtained from terminal 200A to obtain the output text corresponding to the prompt information. Of course, server 200B can store the output text corresponding to the prompt information locally, or send it back to terminal 200A or transmit it to other terminals.

[0092] Terminal 200A and server 200B communicate with each other. For example, terminal 200A and server 200B can communicate via wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0093] The embodiments of this application do not limit the specific technology or device form used in the terminal 200A. For example, the terminal 200A can be a mobile phone (e.g., a foldable phone), a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smartwatch, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, and so on.

[0094] The embodiments of this application do not limit the specific technology or device form used in the server 200B. For example, the server 200B can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, etc.

[0095] It should be understood that the above Figure 2 The application scenarios shown are merely examples and do not constitute any limitation on the application scenarios to which the data processing methods provided in the embodiments of this application are applicable.

[0096] The following is combined with Figure 4 and Figure 5The hardware and software structures of the electronic devices to which the data processing methods provided in the embodiments of this application are applicable will be described in detail. It should be understood that... Figure 4 and Figure 5 The electronic device 100 shown may be Figure 2 The terminal 200A or server 200B shown, or it could also be Figure 3 The mobile phone shown.

[0097] For example, Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0098] Please see Figure 4 The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0099] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0100] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), central processing unit (CPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the central nervous system and command center of electronic device 100. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0101] In one example, processor 110 may include Figure 4 The diagram shows a first processor 110A and a second processor 110B. The first processor 110A contains a target model; that is, the target model resides on the first processor 110A, and the target model can be run by running the first processor. The second processor 110B contains a draft model; that is, the draft model resides on the second processor 110B, and the draft model can be run by the second processor. The specific form of the first processor 110A and the second processor 110B is not limited in this embodiment. For example, the first processor 110A can be an NPU, and the second processor 110B can be a GPU; or, for example, the first processor 110A can be a GPU, and the second processor 110B can be an NPU, etc.

[0102] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0103] For example, in one instance, processor 110 is used to execute any of the data processing methods provided in the embodiments of this application.

[0104] In one example, processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0105] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0106] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0107] Display screen 194 is used to display an always-on charging interface, gallery interface, SMS interface, images, videos, etc. For example, the application interface displayed on display screen 194 may include, but is not limited to, the following: Figure 3The user interface of the mobile phone is shown. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Micro-LED, a quantum dot light-emitting diode (QLED), etc. In one example, the electronic device 100 may include one or N displays 194, where N is an integer greater than 1.

[0108] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions, such as saving music, video, and other files on the external memory card.

[0109] Internal memory 121 can be used to store computer-executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created by electronic device 100 during use (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0110] It should be understood that the above Figure 4 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0111] The hardware system of electronic device 100 has been described in detail above. The software system of electronic device 100 is described below. The software system can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment takes a layered architecture as an example to exemplarily describe the software system of electronic device 100.

[0112] Figure 5 This is a schematic diagram of a software system for an electronic device provided in an embodiment of this application.

[0113] Please see Figure 5 The software system of electronic device 100 adopts a layered architecture. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer 510, application framework layer 520, Android Runtime and core library layer 530, hardware abstraction layer (HAL) 540, and kernel layer 550.

[0114] Application layer 510 may include a series of application packages. For example, Figure 5 The application package shown may include applications such as gallery, SMS, maps, calendar, music, camera, video, navigation, and Bluetooth. For example, the SMS application calls a model located on the processor in the electronic device (including a target model located in the first processor and a draft model located in the second processor), so that the model processes the SMS message based on prompt words and generates corresponding output text (such as schedule information).

[0115] The application framework layer 520 provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer 520 includes some predefined functions. For example... Figure 5 The application framework layer 520 shown includes a window manager, an activity manager, an input manager, and a view system.

[0116] The Android Runtime consists of the core libraries and the virtual machine. The Android Runtime is responsible for scheduling and managing the Android system. The core libraries comprise two parts: one part consists of functionalities that programming languages ​​(such as Java) need to call, and the other part consists of the Android core libraries.

[0117] Application layer 510 and application framework layer 520 run in a virtual machine. The virtual machine executes the programming files (e.g., Java files) of application layer 510 and application framework layer 520 as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0118] The core library layer 530 can include multiple functional modules, such as the surface manager, media framework, libc, and SQLite. The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for various applications. The media framework supports playback and recording of various common audio and video formats, as well as still image files. libc (the C library) is the standard library for the C language and is one of the lowest-level libraries in the system. libc is implemented through Linux system calls. For example, libc can be used to connect or disconnect camera services, set camera shooting parameters, start and stop previews, and take photos.

[0119] HAL 540 is an interface layer located between the operating system kernel and upper-level software, designed to abstract hardware. The Hardware Abstraction Layer (HAL) is an abstract interface for device kernel drivers, providing application programming interfaces (APIs) that allow access to the underlying device to higher-level Java API frameworks. HAL contains multiple library modules, each implementing an interface for a specific type of hardware component. When the system framework layer API requests access to the portable device's hardware, the Android operating system loads the library module for that hardware component. For example, Figure 5 The HAL 540 shown includes a display module, a camera module, and a sensor module.

[0120] Kernel layer 550 is the foundation of the Android operating system; all the final functions of the Android operating system are implemented through the kernel layer. For example, Figure 5 The kernel layer 550 shown includes a display driver, a camera driver, and a gyroscope driver.

[0121] It should be noted that the application provided Figure 5 The illustrated software architecture diagram of the electronic device is merely an example and does not limit the specific module divisions within different layers of the Android operating system. For details, please refer to the descriptions of the Android operating system software architecture in conventional technologies. Furthermore, the data processing method provided in this application can also be based on other operating systems, which will not be listed here.

[0122] The following is combined with Figures 6 to 14 The data processing method provided in the embodiments of this application will be described in detail. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0123] Example 1

[0124] The following is in conjunction with the information provided in this application. Figure 6 The timing diagram of speculative sampling inference shown in Figure (a) illustrates a data processing method provided in this application, which takes the speculative sampling inference performed on two prompt words (one prompt word corresponds to one user task), where the initial target word sequence corresponding to each prompt word is divided into two target word blocks, and the decode phase of the speculative sampling inference for each prompt word includes only one iteration. Figure 7 A flowchart of data processing method 700 is shown. It should be understood that... Figure 6 Figure (a) shows a multi-task reasoning scenario (i.e., two user tasks corresponding to two prompt words).

[0125] For example, Figure 7 This is a schematic diagram of a data processing method 700 provided in an embodiment of this application. The method 700 can be performed by... Figure 2 The terminal 200A and server 200B shown are... Figure 3 The mobile phone shown or Figure 4 The illustrated electronic device 100 performs this action. See also... Figure 7 The method 700 includes steps S710 to S790, which are described below.

[0126] In step S710, the prompt word #A (i.e., the first input text) is input into the target model located on the NPU (i.e., the first processor), and the target model generates the target pre-fill result #A (i.e., the first target pre-fill result). The target pre-fill result #A includes the KV data of the first target token #A and the initial target word sequence A (denoted as the target token sequence #A0) (i.e., the first word sequence) corresponding to the prompt word #A.

[0127] The target model performs step S710 above, that is, it uses the target model located on the NPU to perform the prefill stage (i.e., the first target prefilling process) on the prompt word A. Figure 6 The diagram in (a) shows prefill(TM){jobA}. It should be understood that... Figure 6 In Figure (a), the bold rounded rectangle is the timing diagram corresponding to the cue word #A. Figure 6In Figure (a), the non-bold rounded rectangle is the timing diagram corresponding to the prompt word #B.

[0128] In this embodiment, the target model is located on the NPU of an electronic device (e.g., a mobile phone), meaning the target model is deployed on the NPU of the electronic device. Therefore, the electronic device can enable the target model to perform the above-described step S710 by running the NPU.

[0129] The prompt word #A is a question text that the user inputs into the target model. The prompt word #A is used to request the execution of user task A.

[0130] The first target token #A is the first token in the output text #A corresponding to the prompt word #A. For example, if the output text #A is the token sequence [BHJ], the first target token #A could be token [B].

[0131] The initial target word sequence A is the word sequence obtained after converting the prompt word #A into a token sequence. The initial target word sequence A may include the first target word block A (i.e., the kth first target word block, Q=2 and k=1) and the second target word block A (i.e., the kth first target word block, Q=2 and k=2), wherein each target word block A includes at least one token. It should be understood that the number of words included in the first target word block A may be the same as or different from the number of words included in the second target word block A, and there is no limitation on this. For example, taking the initial target word sequence A as a token sequence [1 2 3 4] as an example, the first target word block A can be a token sequence [1 2], and the second target word block B can be a token sequence [3 4]. For example, taking the initial target word sequence A as the token sequence [1 2 3 4 5] as an example, the first target word block A can be the token sequence [1 2], and the second target word block B can be the token sequence [3 4 5].

[0132] The key-value data of the initial target word sequence A may include each token in the initial target word sequence A. and The matrix (i.e., the key-value pairs of tokens in the initial target lexical sequence A). Optionally, the target model can also cache the KV data of the target token sequence #A0 (i.e., the first lexical sequence) in the target cache space #A for reuse in the subsequent decoding stage corresponding to the prompt word #A. This avoids repeatedly calculating the KV data of the target token sequence #A0, reducing computational overhead and improving computational efficiency.

[0133] The target model executes step S710 above, that is, inputting the prompt word #A into the target model located on the NPU, and the target model generates the target pre-filled result #A. Exemplarily, this may include the following steps: the target model performs data preprocessing on the prompt word #A (e.g., including lexical transformation processing and block processing) to obtain the target token sequence #A0; the target model calculates the KV data for each target lexical block A in the target token sequence #A0; the target model concatenates the KV data of the second target lexical block A after the KV data of the first target lexical block A to obtain the KV data of the target token sequence #A0; the target model processes the KV data of the target token sequence #A0, as well as the token at the last position in the target token sequence #A0... Attention is calculated on the matrix to obtain the first target token #A.

[0134] In step S720, the prompt word #B (i.e., the second input text) is input into the target model, and the target model generates the target pre-filled result #B (i.e., the second target pre-filled result). The target pre-filled result #B includes the KV data of the first target token #B and the initial target word sequence B (denoted as the target token sequence #B0) (i.e., the second word sequence) corresponding to the prompt word #B.

[0135] The target model performs step S720 above, that is, it uses the target model located on the NPU to perform the prefill stage (i.e., the second target prefilling process) on the prompt word B. Figure 6 The diagram in (a) shows prefill(TM){jobB}.

[0136] In this embodiment, the target model is located on the NPU of an electronic device (e.g., a mobile phone), meaning the target model is deployed on the NPU of the electronic device. Therefore, the electronic device can enable the target model to perform the above-described step S720 by running the NPU.

[0137] The prompt word #B is another question text that the user inputs into the target model. The prompt word #B is used to request the execution of user task B.

[0138] It should be understood that user task B corresponding to prompt word #B and user task A corresponding to prompt word #A are two different user tasks. There are no restrictions on the task type of user task B and the task type of user task A. In one example, the task type of user task B is different from that of user task A. For example, user task A can be a text generation or text translation task, or a query task. In another example, the task type of user task B is the same as that of user task A. For example, user task A is an image recognition task for image 1, and user task B is an image recognition task for image 2.

[0139] The first target token #B is the first token in the output text #B corresponding to the prompt word #B.

[0140] The KV data of the initial target word sequence B includes the tokens in the initial target word sequence B. and Matrix. Optionally, the target model can also cache the KV data of the target token sequence #B0 in the target cache space #B for reuse in the decoding stage corresponding to the prompt word #B. This avoids repeatedly calculating the KV data of the target token sequence #B0, reducing computational overhead and improving computational efficiency.

[0141] The initial target word sequence B is the sequence obtained after converting the prompt word #B into a token sequence. The initial target word sequence B includes the first target word block B (i.e., the k2nd second target word block, Q2=2, and k2=1) and the second target word block B (i.e., the k2nd second target word block, Q2=2, and k2=2), where each target word block B includes at least one word element. It should be understood that the number of tokens included in the first target word block B can be the same as or different from the number of tokens included in the second target word block B; this is not limited.

[0142] Target cache space #B and target cache space #A are spaces that the target model can access. For example, target cache space #B and target cache space #A can be located in the target model.

[0143] It should be understood that the working principle of the target model performing the above step S720 is the same as the working principle of the target model performing the above step S710. For the content not described here, please refer to the description in the above step S710.

[0144] The order in which steps S710 and S720 are executed on the target model is not limited. For example, the target model can execute step S720 first and then step S710.

[0145] Optionally, the target model can also perform a step to obtain prompt words, and the method by which the target model obtains prompt words is not limited. For example, a user can input prompt words through a front-end interface, allowing the target model to obtain prompt words #A and #B. Alternatively, an engineer can input prompt words through code, allowing the target model to obtain prompt words #A and #B. Prompt words #A and #B can correspond to different users or the same user.

[0146] In step S730, the initial draft lexical sequence A (denoted as draft token sequence #A0) is input into the draft model located on the GPU. The draft model generates a draft pre-filling result #A (i.e., the first draft pre-filling result). The draft pre-filling result #A includes the KV data of the first candidate token #A and the initial draft lexical sequence A. The initial draft lexical sequence A is a sequence determined based on the target token sequence #A0 and the first target token #A.

[0147] The draft model performs the above step S730, that is, it uses the draft model located on the GPU to perform the prefill stage (i.e., the first draft prefilling process) on the prompt word #A. Figure 6 The diagram (a) illustrates prefill(DM){jobA}. In this embodiment, the draft model resides on the GPU of an electronic device (e.g., a mobile phone), meaning the draft model is deployed on the GPU of the electronic device. Therefore, the electronic device can enable the draft model to perform the above-described step S730 by running the GPU.

[0148] Optionally, the GPU in step S730 above can be replaced with other more efficient computing units, and there are no restrictions on other computing units.

[0149] There are no restrictions on how the electronic device runs the draft model on the GPU or the target model on the NPU. For example, the electronic device starts a main thread to run the target model on the NPU, while simultaneously starting a sub-thread to run the draft model on the GPU.

[0150] The initial draft lexical sequence A is a sequence determined based on the target token sequence #A0 and the first target token #A. The initial draft lexical sequence A includes the first draft lexical block A (i.e., the kth first draft lexical block, Q=2 and k=1) and the second draft lexical block A (i.e., the kth first draft lexical block, Q=2 and k=2). For details not elaborated here, please refer to the description in the previous example "The initial draft lexical sequence is determined based on the initial target lexical sequence and the first target token".

[0151] Optionally, the draft model can also perform the step of caching the KV data of the draft token sequence #A0 into the draft cache space #A, so that the draft model can use it in the subsequent decoding stage corresponding to the prompt word #A.

[0152] Optionally, before performing step S730 above, the draft model may also perform the step of obtaining the draft token sequence #A0. The step of obtaining the draft token sequence #A0 by the draft model is not limited. For example, the draft model may interact with the target model to obtain the target token sequence #A0 and the first target token #A from the target model; the draft model obtains the initial draft lexical sequence A based on the target token sequence #A0 and the first target token #A. For example, the draft model may interact with the target model to obtain the initial draft lexical sequence A from the target model.

[0153] In this embodiment, neither the type of the draft model nor the type of the target model is limited. For example, the draft model may be, but is not limited to, a small LLM or an n-gram model. For example, the target model may be used in applications such as natural language processing, image processing, or speech recognition and processing, including but not limited to LLM, large computer vision (CV) models, etc., and the type of the target model is not limited.

[0154] It should be understood that in the implementation of steps S710 and S730 provided in the embodiments of this application, the target model located on the NPU executes step S710 first, and then the draft model located on the GPU executes step S730. That is, the prefill stage of the initial target lexical sequence A corresponding to prompt word A# (denoted as the prefill stage of the target model corresponding to prompt word #A) executed by the target model on the NPU, and the prefill stage of the initial draft lexical sequence A corresponding to prompt word A# executed by the draft model on the GPU (denoted as the prefill stage of the draft model corresponding to prompt word #A) are two stages executed sequentially. In other words, the prefill stage of the target model corresponding to prompt word #A and the prefill stage of the draft model corresponding to prompt word #A do not overlap in time.

[0155] In step S740, the initial draft lexical sequence B (denoted as draft token sequence #B0) corresponding to the prompt word #B is input into the draft model, and the draft model generates a draft pre-filling result #B (i.e., the second draft pre-filling result). The draft pre-filling result #B includes the KV data of the first candidate token #B and the initial draft lexical sequence B. The initial draft lexical sequence B is a sequence determined based on the token sequence #B0 and the first target token #B.

[0156] The draft model performs step S740 above, that is, it uses the draft model located on the GPU to perform the prefill stage for the prompt word #B, as described above. Figure 6 The diagram in (a) shows prefill(DM){jobB}.

[0157] The initial draft lexical sequence B is a sequence determined based on the token sequence #B0 and the first target token #B. The initial draft lexical sequence B includes the first draft lexical block B and the second draft lexical block B. For details not elaborated here, please refer to the description in the previous example "The initial draft lexical sequence is determined based on the initial target lexical sequence and the first target token".

[0158] Optionally, the draft model can also perform the step of caching the KV data of the draft token sequence #B0 into the draft cache space #B, so that it can be used by the draft model in the subsequent decoding stage corresponding to the prompt word #B. Draft cache space #B and draft cache space #A can be two different cache spaces, and the draft model can access both of them.

[0159] The principle of performing step S740 on the draft model is the same as that of performing step S730 on the draft model. For details not elaborated here, please refer to the relevant description in step S730 above.

[0160] It should be understood that in the implementation of steps S720 and S740 provided in the embodiments of this application, the target model located on the NPU executes step S720 first, and then the draft model located on the GPU executes step S740. That is, the prefill stage of the initial target lexical sequence B corresponding to prompt word B# (denoted as the prefill stage of the target model corresponding to prompt word #B) executed by the target model on the NPU and the prefill stage of the initial draft lexical sequence B corresponding to prompt word B# executed by the draft model on the GPU (denoted as the prefill stage of the draft model corresponding to prompt word #B) are two stages executed sequentially. In other words, the prefill stage of the target model corresponding to prompt word #B and the prefill stage of the draft model corresponding to prompt word #B do not overlap in time.

[0161] There is no restriction on the order of performing steps S730 and S740 on the draft model. For example, the draft model can perform step S740 first and then step S730.

[0162] At this point, the electronic device has completed steps S710 to S740, that is, the electronic device has completed the prefill stage corresponding to prompt word #A and the prefill stage corresponding to prompt word #B. See also Figure 6 In Figure (a), the calculation time required for the electronic device to perform the above steps S710 to S740 is t0.

[0163] The following sections, in conjunction with steps S750 to S780, describe the decoding stage (i.e., the first iteration of round N) corresponding to the prompt word #A and the decoding stage (i.e., the second iteration of round M) corresponding to the prompt word #B.

[0164] In step S750, the draft model generates a first-round candidate word sequence #A1 (the i-th round candidate word, where i=1) based on the draft token sequence #A1. The draft token #A1 sequence is obtained by continuating the first candidate token #A after the draft token #A0 sequence. The first-round candidate word sequence #A1 includes R1 candidate words #A1, where R1 is a positive integer.

[0165] It should be understood that the process of executing step S750 above involves using the draft model located on the GPU to perform the l-th round of decoding on the prompt word A (i.e., the i-th round of first draft decoding operation, where i=1), which is the above... Figure 6 The process shown is guess(DM)_1{jobA}.

[0166] The draft model executes step S750 above, that is, the draft model uses an autoregressive generation mechanism to generate R1 candidate tokens #A1 one by one based on the draft token sequence #A1. The autoregressive generation mechanism, which is not described in detail here, can be found in the relevant description above. It should be understood that the draft model needs to use the KV data of the draft token #A0 sequence during the execution of step S750 above, which can be obtained by the draft model from the draft cache space #A.

[0167] In step S760, the target model performs parallel verification on the candidate word sequence #A1 in the first round based on the target token #A1 sequence, generating the first verification word sequence #A1 (i.e. the first iteration result in the i-th round, and i=1). The target token #A1 sequence is the sequence obtained by concatenating the first target token #A after the target token sequence #A0.

[0168] It should be understood that the process of the target model executing the above step S760 is that it uses the target model located on the NPU to perform the l-th round of decoding on the prompt word A (i.e., the i-th round of the first target decoding operation, and i=1), which is the above. Figure 6 The decode(TM)_1{jobA} process is shown.

[0169] There are no restrictions on the first round of verification token sequence #A1. For example, when each candidate token #A1 in the first round of candidate token sequence #A1 passes verification, the first round of verification token sequence #A1 can be a sequence obtained by continuating the first round of candidate token sequence #A1 after the first target token #A.

[0170] The parallel verification method for the target model in performing step S760 is the same as the existing method and is not limited thereto. For example, the target model performing step S760 may include the following steps: the target model constructs Q input sequences #A1 based on the target toQen#A1 sequence and the first-round candidate lexical sequence #A1; the target model processes the Q input sequences #A1 to obtain Q inference results #A1, where one input sequence #A1 corresponds to one inference result #A1, and one inference result #A1 corresponds to one inference toQen; the target model uses the Q inference results #A1 to verify the first-round candidate lexical sequence #A1, obtaining the first-round verified lexical sequence #A1.

[0171] In the above steps of constructing multiple input sequences #A1 based on the target toQen#A1 sequence and the first round candidate word sequence #A1, the following steps can be included, for example: the target model constructs Q initial input sequences #A1 based on Q candidate words #A1, wherein the h-th initial input sequence among the Q initial input sequences includes the first h candidate words in the Q candidate words #A1, h=1,......,Q; the target model concatenates each initial input sequence #A1 after the target toQen#A1 sequence to obtain the corresponding input sequence #A1, so as to obtain Q input sequences #A1. For example, taking the target toQen#A1 sequence as [12 3] and the Q (Q=3) candidate word units #A1 as [5 6 7], the Q initial input sequences #A1 can include the following 3 sequences: [5], [5 6], [5 6 7], and the Q input sequences #A1 can include the following 3 sequences: [1 2 3 5], [1 2 3 5 6], [1 2 3 5 6 7].

[0172] In the process of the target model processing Q input sequences #A1 to obtain Q inference results #A1, the target model needs to utilize the QV data of the target toQen sequence #A0, which can be obtained by the target model from the target cache space #A.

[0173] In the step of the above target model verifying the first round candidate lexical sequence #A1 using Q inference results #A1 to obtain the first round verification lexical sequence #A1, exemplarily, the following steps may be included: if the p-th inference result #A1 among the Q inference results #A1 is the same as the p-th candidate lexical #A1 among the Q candidate lexical #A1, and p=1,......,Q, the target model will determine the first round candidate lexical sequence #A1 as the first round verification lexical sequence #A1, that is, the first round verification lexical sequence #A1. A1 includes Q candidate words #A1; if the q-th inference result #A1 in the Q inference results #A1 is different from the p-th candidate word #A1 in the Q candidate words #A1, and q is an integer less than or equal to Q, the target model will append the q-th inference result #A1 to the partial candidate words #A1 to obtain the sequence, which is determined as the first round of verification word sequence #A1. Among them, the partial candidate words #A1 include the candidate words #A1 in the Q candidate words #A1 that are located before the position of the p-th candidate word #A1. For example, taking Q inference results #A1 as [1 2 3] and Q candidate words #A1 as [ABC], if the first inference result #A1[1] is the same as the first candidate word #A1[A], the second inference result #A1[2] is the same as the second candidate word #A1[B], and the third inference result #A1[3] is the same as the third candidate word #A1[C], the target model will determine the first round of verification word sequence #A1 based on [ABC]. For example, taking Q inference results #A1 as [1 2 3] and Q candidate words #A1 as [ABC], if the first inference word [1] is the same as the first inference word #A1 [A], the second inference #A1 [2] is the same as the second inference word #A1 [B], and the third inference #A1 [3] is different from the third inference word #A1 [C], the target model will determine the first round of verification word sequence #A1 based on [AB 3].

[0174] Optionally, before performing step S760 above, the target model may also perform the step of obtaining the target token sequence #A1. The step of obtaining the target token sequence #A1 is not limited. For example, obtaining the target token sequence #A1 may include the following steps: the target model obtains the first round candidate word sequence #A1 from the draft model; the target model sequentially appends the first target token #A and the first round candidate word sequence #A1 to the target token sequence #A0 to obtain the target token sequence #A1.

[0175] In step S770, the draft model generates the first round candidate word sequence #B1 (i.e., the j-th round candidate word, where j=1) based on the draft token sequence #B1. The draft token #B1 sequence is obtained by concatenating the first candidate token #B after the draft token #B0 sequence. The time period during which the target model on the NPU generates the first round verification word sequence #A1 (hereinafter referred to as time period A1) overlaps with the time period during which the draft model on the GPU generates the first round candidate word sequence #B1 (hereinafter referred to as time period B1). The first round candidate word sequence #B1 includes R2 candidate words #B1, where R2 is a positive integer.

[0176] It should be understood that the draft model located on the GPU executes the process of step S770 above, that is, it uses the draft model located on the GPU to perform the l-th round of decoding stage (i.e., the j-th round of second draft decoding operation, and j=1) on the prompt word B#, which is the above. Figure 6 The process shown is guess(DM)_1{jobB}.

[0177] There is no limitation on the overlapping time period in step S770 above.

[0178] In one example, when the length of time period A1 is greater than the length of time period B1, the overlapping time period can be time period B1 or a portion of time period B1. In this example, the goal of the target model masking the inference latency of the draft model can be achieved.

[0179] In another example, when the length of time period A1 is less than or equal to the length of time period B1, the overlapping time period can be time period A1 or a portion of time period A1. This approach allows the draft model to mask the inference latency of the target model.

[0180] The draft model executes step S770 above, that is, the draft model uses an autoregressive generation mechanism to generate R2 candidate tokens #B1 one by one based on the draft token sequence #B1. The autoregressive generation mechanism, which is not described in detail here, can be found in the relevant description above. It should be understood that the draft model needs to use the KV data of the draft token #B0 sequence during the execution of step S770 above, which can be obtained by the draft model from the draft cache space #B.

[0181] In step S780, the target model performs parallel verification on the first round candidate word sequence #B1 based on the target token #B1 sequence, generating the first round verification word sequence #B1 (i.e. the second iteration result of the jth round, j=1), where the target token #B1 sequence is the sequence obtained by continuating the first target token #B after the target token sequence #B0.

[0182] It should be understood that the process of executing step S780 above is to use the target model located on the NPU to perform the l-th round of decoding on the prompt word #B (i.e., the j-th round of second target decoding operation, where j=1), which is the above. Figure 6 The decode(TM)_1{jobB} process is illustrated in (a) of the diagram.

[0183] There are no restrictions on the first round of verification token sequence #B1. For example, when each candidate token #B1 in the first round of candidate token sequence #B1 passes verification, the first round of verification token sequence #B1 can be a sequence obtained by continuating the first round of candidate token sequence #B1 after the first target token #B.

[0184] The principle behind the target model executing step S780 is the same as that of the target model executing step S760. Details not elaborated here can be found in the description of step S760. It should be understood that the target model needs to utilize the KV data of the target token sequence #B0 during the execution of step S780, which can be obtained from the target cache space #B.

[0185] The calculation time for executing steps S750 to S780 above, i.e., the calculation time for executing the decoding phase of prompt word #A and the decoding phase of prompt word #B, is as follows: Figure 6 The calculation time t1 is shown in Figure (a).

[0186] It should be understood that in the above-described decoding stage provided in the embodiments of this application, the example is that the electronic device first executes steps S750 and S760 (i.e., the first round of decoding corresponding to prompt word #A) sequentially, and then executes steps S770 and S7780 (i.e., the first round of decoding corresponding to prompt word #B) sequentially. Optionally, the electronic device may first execute steps S770 and S780 (i.e., the first round of decoding corresponding to prompt word #B) sequentially, and then execute steps S750 and S760 (i.e., the first round of decoding corresponding to prompt word #A) sequentially, see [link to relevant documentation]. Figure 6 The timing diagram for this implementation shown in Figure (b) is... Figure 6 In Figure (b), the time interval during which the target model on the NPU executes decode(TM)_1{jobB} overlaps with the time interval during which the draft model on the GPU executes guess(DM)_1{jobA}. Therefore, this overlapping time interval can be saved, reducing computational latency, accelerating model inference, and improving processing efficiency. It should be understood that... Figure 6 The bold box in Figure (b) is the timing diagram corresponding to the cue word #A. Figure 6 The non-bold box in Figure (b) is the timing diagram corresponding to the prompt word #B.

[0187] In step S790, when the preset iteration condition #A is met, the target model generates output text #A (i.e., the first output text) based on the verification results of each round corresponding to the prompt word #A; and when the preset iteration condition #B is met, the target model generates output text #B (i.e., the second output text) based on the verification results of each round corresponding to the prompt word #B.

[0188] The preset iteration condition #A is used to indicate the stop execution of user task A corresponding to the prompt word #A.

[0189] The preset iteration condition #B is used to indicate the user task B corresponding to the prompt word #B to stop execution.

[0190] because Figure 6 The decode stage shown in (a) only includes one round of iteration calculation corresponding to the prompt word #A. Therefore, the target model outputs the first round of verification word sequence #A1 as the output text #A.

[0191] because Figure 6 The decode stage shown in (a) only includes one round of iteration calculation corresponding to the prompt word #B. Therefore, the target model outputs the first round of verification word sequence #B1 as the output text #B.

[0192] There are no restrictions on preset iteration conditions #A and B#. For example, either preset iteration condition #A or preset iteration condition B# could be that the current iteration time meets the preset iteration duration. Alternatively, either preset iteration condition could be that the current iteration count meets the preset iteration count. Another example is that the length of the currently generated token meets the preset length. Yet another example is that the currently generated token is EOS, etc.

[0193] It should be noted that the above steps S790 are all described using the decoding stage as an example, including one round of iterative calculation corresponding to prompt word #A and one round of iterative calculation corresponding to prompt word #B.

[0194] Optionally, when the decoding stage includes multiple rounds of iterative calculations corresponding to the prompt words, that is, when the target model generates multiple rounds of verification results corresponding to multiple rounds of iterative calculations (one round of iteration corresponds to one round of verification result), and the preset iteration conditions are met, the target model concatenates the multiple rounds of verification results sequentially according to the generation time order to obtain the concatenated result. Then, the target model outputs the concatenated result as the output text.

[0195] It should be understood that the above Figure 7The methods shown are for illustrative purposes only and do not constitute any limitation on the methods provided in the embodiments of this application.

[0196] It should be noted that the above Figure 7 The methods provided are as described above. Figure 6 The time series diagram of speculative sampling inference shown in Figure (a) will be used as an example for introduction. Figure 6 The decode phase shown in (a) includes one iteration for the prompt word #A and one iteration for the prompt word #B.

[0197] Optionally, based on the above-described embodiments provided in this application... Figure 6 The decode stage shown in diagram (a) can also yield... Figure 8 The decode stage is shown. See also Figure 8 The decode phase includes two iterations corresponding to prompt #A and one iteration corresponding to prompt #B. The decode phase includes an overlapping time period #1 (i.e., at least one first overlapping time period). This overlapping time period #1 is the time period during which the target model on the NPU executes decode(TM)_1{jobA} (i.e., the first iteration of the target model corresponding to prompt #A), and the time period during which the draft model on the GPU executes guess(DM)_1{jobB} (i.e., the first iteration of the draft model corresponding to prompt #B). Therefore, it is also possible to base the above... Figure 7 The working principle of steps S750 to S780 in the provided data processing method is as follows: Figure 8 The data processing methods shown in the decode stage will not be elaborated here.

[0198] Optionally, based on the above-described embodiments provided in this application... Figure 6 The decode stage shown in diagram (a) can also yield... Figure 9 The decoding stage is illustrated in diagram (a) above. See also... Figure 9In Figure (a), the decode phase includes two iterations corresponding to prompt #A and one iteration corresponding to prompt #B. The decode phase includes two overlapping time periods #1 (i.e., at least one first overlapping time). The first overlapping time period #1 is the time period during which the target model on the NPU executes decode(TM)_1{jobA} (i.e., the first iteration of the target model corresponding to prompt #A), and the time period during which the draft model on the GPU executes guess(DM)_1{jobB} (i.e., the first iteration of the draft model corresponding to prompt #B). The second overlapping time period #1 is the time period during which the target model executes decode(TM)_1{jobB} (i.e., the first iteration of the target model corresponding to prompt #B), and the time period during which the draft model executes guess(DM)_2{jobA} (i.e., the second iteration of the draft model corresponding to prompt #A). Therefore, it is also possible to base the above... Figure 7 The working principle of steps S750 to S780 in the provided data processing method is as follows: Figure 9 The data processing method in the decode stage, illustrated in (a) above, will not be described in detail here.

[0199] Optionally, based on the above-described embodiments provided in this application... Figure 6 The decode stage shown in diagram (a) can also yield... Figure 9 The decoding stage is illustrated in diagram (b) of the diagram. See also... Figure 9 In Figure (b), the decode phase includes two iterations corresponding to cue word #A and two iterations corresponding to cue word #B. The decode phase includes three overlapping time periods #1 (i.e., at least one first overlapping time period). The first two overlapping time periods #1 of these three overlapping time periods #1 are... Figure 9 The two overlapping time periods #1 shown in Figure (a) are the same. The third overlapping time period #1 of these three overlapping time periods #1 is the time period during which the target model executes decode(TM)_2{jobA} (i.e., the second iteration of the target model corresponding to prompt #A), and overlaps with the time period during which the draft model executes guess(DM)_2{jobB} (i.e., the second iteration of the draft model corresponding to prompt #B). Therefore, it is also possible to base this on the above... Figure 7 The working principle of steps S750 to S780 in the provided data processing method is as follows: Figure 9 The data processing method in the decode stage, as shown in diagram (b) in the figure, will not be described in detail here.

[0200] In this embodiment, in a multi-task recommendation scenario, for each task (e.g., job A corresponding to prompt #A and prompt #B) of the two tasks (i.e., job A and job B) corresponding to two prompt words (i.e., prompt #A and prompt #B), the pre-filling stage and the decoding stage of each task (e.g., job A corresponding to prompt #A) are executed sequentially; and for the two tasks (i.e., job A and job B), the decoding stage of one task (job A) and the decoding stage of the other task (job B) are executed in parallel. Specifically, by setting the processing time of the target model running the decoding stage of one task (job A) in one processor (i.e., NPU) to overlap with the processing time of the draft model running the decoding stage of another task (job B) in another processor (i.e., GPU), the goal of parallel execution of the decoding stage of one task (job A) by the target model in one processor (i.e., NPU) and the decoding stage of another task (job B) by the draft model in another processor (i.e., GPU) can be achieved. Compared to traditional speculative sampling, where both the target model and draft model are deployed on a single processor, and the decoding processes of the target model on that processor for one task (job A) and the target model on that processor for another task (job B) are executed sequentially, the solution provided in this application decouples the target model and draft model by deploying them on two different processors. Since the first processor is better suited to the more complex target model, and the second processor is better suited to the simpler draft model, decoupling the target model and draft model improves the computational efficiency and resource utilization of each processor. Furthermore, by setting the processing time of the target model running the decoding phase of one task (job A) on one processor (i.e., the NPU) to overlap with the processing time of the draft model running the decoding phase of another task (job B) on another processor (i.e., the GPU), this overlapping time can be saved, reducing computational latency, accelerating model inference, and thus improving processing efficiency.

[0201] Example 2

[0202] The following describes a multi-task recommendation scenario: Speculative sampling inference is performed on two prompt words (one prompt word corresponds to one user task). The initial target word sequence corresponding to each prompt word is divided into two target word blocks. The prefill stage corresponding to prompt word #A is executed first, followed by the prefill stage corresponding to prompt word #B. This paper introduces another data processing method provided in the embodiments of this application. Figure 10 A flowchart of data processing method 1000 is shown. It should be understood that... Figure 10 The data processing method in the prefill stage of the provided method 1000 is based on the above. Figure 7 The optimization of the prefill stage shown in steps S710 to S740 of the provided embodiment one differs in that, in the above embodiment one, the prefill stage of the target model corresponding to prompt word #A and the prefill stage of the draft model corresponding to prompt word #A are executed serially; and the prefill stage of the target model corresponding to prompt word #B and the prefill stage of the draft model corresponding to prompt word #B are also executed serially; while in embodiment two, the prefill stage of the target model corresponding to prompt word #A and the prefill stage of the draft model corresponding to prompt word #A are executed in parallel; and the prefill stage of the target model corresponding to prompt word #B and the prefill stage of the draft model corresponding to prompt word #B are also executed in parallel.

[0203] For example, Figure 10 This is a schematic diagram of a data processing method 1000 provided in an embodiment of this application. The method 1000 can be performed by... Figure 2 The terminal 200A and server 200B shown are... Figure 3 The mobile phone shown or Figure 4 The illustrated electronic device 100 performs this action. See also... Figure 10 The method 1000 includes steps S1010a to S1040a (i.e., the prefill stage corresponding to the prompt word for the target model on the NPU), and steps S1010b to S1040b (i.e., the prefill stage corresponding to the prompt word for the draft model on the GPU).

[0204] In step S1010a, the target model located on the NPU (i.e., the first processor) prefills the kth target word block A (i.e. the kth first target word block) among the Q target word blocks A (i.e. the Q first target word blocks) included in the initial target word block sequence A (i.e. the first word block sequence corresponding to the first input text), to obtain the KV data of the kth target word block A (i.e. the kth first target block result), where Q is a positive integer greater than or equal to 2.

[0205] The initial target word sequence A is the sequence obtained after converting the prompt word #A (i.e. the first input text) into a token sequence. The initial target word sequence A includes Q target word blocks A, each of which includes at least one token. Q is a positive integer greater than or equal to 2, and there is no restriction on the value of Q.

[0206] The k-th target word block A is any one of the Q target word blocks A, where k = 1, 2, ..., Q.

[0207] For example, when Q equals 2 and k equals 1, the above step S1020a is executed, i.e. Figure 11 The process shown in prefill(TM){jobA} is the prefill process for the first target word block A.

[0208] For example, when Q equals 2 and k equals 2, the above step S1020a is executed, i.e. Figure 11 The process of performing prefill on the second target word block A is shown in prefill(TM){jobA}.

[0209] Optionally, the process of prefilling the initial target lexical sequence A by the target model may also include a data preprocessing step 1, which is used to convert the prompt word #A into the initial target lexical sequence A. The implementation of data preprocessing step 1 is not limited, nor is its name.

[0210] In step S1020a, the target model determines whether k is equal to M.

[0211] After the target model executes step S1020a, if it determines that k equals Q, the target model continues to execute step S1030a; if it determines that k is not equal to (i.e. less than) M, the target model sets k=k+1 and returns to execute step S1010a.

[0212] In step S1030a, the target model generates target pre-filling result #A (i.e., the first target pre-filling result) based on the KV data of the Q target word blocks A corresponding to the Q target word blocks A one-to-one. The target pre-filling result #A includes the KV data of the first target token #A and the initial target word sequence A.

[0213] There is a one-to-one correspondence between the KV data of the Q target word blocks A and the Q target word blocks A, that is, the k-th target word block A corresponds to the KV data of the k-th Q target word blocks A.

[0214] The target model performs the above step S1030a, which may include the following steps, for example: The target model concatenates the KV data of Q target word blocks A according to the time order of the generation of the KV data of the kth target word block A, to generate the KV data of the initial target word sequence A; The target model performs attention calculation on the KV data of the Q target word blocks A and the Q matrix of the token at the last position in the Qth target word block A to obtain the first target token #A, wherein the Q matrix of the token at the last position in the Qth target word block A is the Q matrix of the token at the last position in the initial target word sequence A.

[0215] Optionally, the target model can also perform the step of caching the KV data of the initial target lexical sequence A into the target cache space #A, so that it can be reused in the subsequent decoding stage corresponding to the prompt word #A.

[0216] It should be understood that the above steps S1010a to S1030a are the chunk prefill stage (i.e., the first target prefill process) corresponding to the target model execution prompt word #A on the NPU, and the above steps S1010a to S1030a are a serial execution process.

[0217] In step S1040a, the target model performs the prefill stage (i.e., the second prefill process) corresponding to the prompt word #B (i.e. the second output text).

[0218] The target model performs the above step S1040a, that is, the target model processes the initial target word sequence B (denoted as the target token sequence #B0) corresponding to the prompt word #B to generate the target pre-filling result #B (i.e. the second pre-filling result), wherein the target pre-filling result #B includes the KV data of the first candidate token #B and the target token sequence #B0.

[0219] Optionally, the target model can also cache the KV data of the target token sequence #B0 in the target cache space #B for use in the decoding stage corresponding to the subsequent prompt word #B.

[0220] It should be understood that the working principle of the target model performing the above steps S1040a is the same as that of the target model performing the above steps S1010a to S1030a, and will not be described in detail here.

[0221] In step S1010b, the draft model prefills the kth draft word block A (i.e., the kth first draft word block) in the initial draft word block sequence A (i.e., the i-th round of draft data, i=1) to obtain the KV data of the kth draft word block A (i.e., the kth first draft block result). The time period during which the draft model prefills the w1th draft word block A (denoted as time period B2) overlaps with the time period during which the target model prefills the w2th target word block A (denoted as time period A2). The initial draft word block sequence A includes Q draft word blocks A that correspond one-to-one with the Q target word blocks A. w2 is a positive integer greater than or equal to w1 and less than Q, and w1 is a positive integer less than or equal to Q.

[0222] The initial draft lexical sequence A is a lexical sequence determined based on the initial target lexical sequence A and the first target token #A. For details not elaborated here, please refer to the relevant description above that "the initial draft lexical sequence is a lexical sequence determined based on the initial target lexical sequence and the first target token".

[0223] The initial draft lexical sequence A consists of Q draft lexical blocks A. That is, dividing the initial draft lexical sequence A into blocks yields Q draft lexical blocks A. Each of the Q draft lexical blocks A may contain the same or different number of lexical units.

[0224] The overlapping time period in step S1010b above is not limited. For example, when the length of time period A2 is greater than the length of time period B2, the overlapping time period can be time period B2, or a portion of time period B2. For example, when the length of time period A2 is less than or equal to the length of time period B2, the overlapping time period can be time period A2, or a portion of time period A2.

[0225] It should be understood that the prefilling of the w1th draft word block A by the draft model is a block prefilling process of the draft model corresponding to the prompt word #A. The prefilling of the w2th target word block A by the target model is a block prefilling process of the target model corresponding to the prompt word #A.

[0226] In step S1010b above, the time period during which the draft model prefills the w1st draft word block A overlaps with the time period during which the target model prefills the w2th target word block A. Here, w2 is a positive integer greater than or equal to w2 and less than Q, and w1 is a positive integer less than or equal to Q. The values ​​of w1 and w2 are not limited.

[0227] In one example, w1 = w2 - 1, which is the time period during which the draft model prefills the (w2 - 1)th draft word block A (i.e., the (w - 1)th first draft word block) (i.e., the processing time during which the draft model generates the (w - 1)th first draft block result based on the (w - 1)th first draft word block), overlaps with the time period during which the target model prefills the w2th target word block A (i.e., the wth first target word block) (i.e., the processing time during which the target model generates the wth first target fill result based on the wth first target word block).

[0228] In this example, the target model can mask the inference latency of the draft model. Specifically, after the target model completes the inference (decode) of the first target word block A, it begins inference of the second target word block A, while simultaneously, the draft model begins inference of the first draft word block A. After the target model completes the inference (decode) of the second target word block A, it begins inference of the third target word block A, while simultaneously, the draft model begins inference of the second draft word block A. And so on, until the target model completes the inference (decode) of the (Q-1)th target word block A, it begins inference of the Qth target word block A, while simultaneously, the draft model begins inference of the (Q-1)th draft word block A. Therefore, the decoding time of the first draft word block A in the draft model can be fully or partially masked by the inference latency (either entirely or partially) of the second target word block A in the target model; the inference time of the second draft word block A in the draft model can be fully or partially masked by the inference latency (either entirely or partially) of the third target word block A in the target model; and so on. The time period for the draft model to prefill the (Q-1)th draft word block A overlaps with the time period for the target model to prefill the Qth target word block A. Therefore, based on the above implementation method of parallel prefilling of the target model corresponding to the prompt word #A with the prefilling of the draft model corresponding to the prompt word #A, the prefilling stage latency of the draft model can be saved, accelerating model inference and improving data processing efficiency.

[0229] For example, let's take Q = 2, w1 = 1, and w2 = 2 as an example. Figure 11 The diagram shows the time period during which the target model prefills the second target word block A corresponding to the prompt word #A, and the time period during which the draft model prefills the first draft word block A corresponding to the prompt word #A. These two time periods overlap, with a time period #2. Figure 6 Compared to the computation time t0 required for the prefill stage corresponding to the prompt word #A shown in diagram (a), Figure 11The prompt word #A indicates that the computation time t2 required for the prefill stage is less than t0, thus reducing computation time and improving data processing efficiency.

[0230] In another example, w1 is any positive integer greater than or equal to w2 and less than Q, and w2 is any positive integer less than or equal to Q. For example, when Q equals 3, w2 can equal 3 and w1 can equal 1. For example, when Q equals 5, w2 can equal 2 and w1 can equal 1.

[0231] It should be understood that, in the embodiments of this application, the prefilling stage of the draft model for the w1st draft word block A and the prefilling stage of the target model for the w2th target word block A are executed in parallel; the prefilling stage of the target model for Q target word blocks A is executed serially, and the prefilling stage of the draft model for Q draft word blocks A is executed serially.

[0232] In step S1020b, the draft model determines whether k is equal to Q.

[0233] After the draft model executes step S1020b, if it is determined that k equals Q, the draft model continues to execute step S1030b; if it is determined that k is not equal to (i.e. less than) Q, the draft model sets k=k+1 and returns to execute step S1010b.

[0234] In step S1030b, the draft model generates a draft pre-filling result #A (i.e., the first draft pre-filling result) based on the KV data of Q draft lexical blocks A. The draft pre-filling result #A includes the KV data of the first candidate token #A and the initial draft lexical sequence A.

[0235] The draft pre-fill result #A in step S1030b above is the same as the draft pre-fill result #A in step S730 above. For details not elaborated here, please refer to the relevant description in step S730 above.

[0236] Optionally, the draft model can also perform the step of caching the KV data of the initial draft lexical sequence A into the draft cache space #A, so that the draft model can use it in the subsequent decoding stage corresponding to the prompt word #A.

[0237] The working principle of the draft model performing the above step S1030b is the same as that of the target model performing the above step S1030a, and will not be repeated here.

[0238] It should be understood that the above steps S1010b to S1030b are the chunk prefill stage (i.e., the first draft prefill process) corresponding to the prompt word #A on the draft model located on the GPU (i.e. the second processor), and the above steps S1010b to S1030b are executed serially.

[0239] In step S1040b, the draft model performs the prefill stage (i.e., the second prefill process) corresponding to the prompt word #B (i.e. the second input text).

[0240] Perform the above step S1040b, that is, the draft model processes the initial draft lexical sequence B (denoted as the draft token sequence #B0) (i.e. the second lexical sequence) corresponding to the prompt word #B to generate the draft pre-fill result #B (i.e. the second draft pre-fill result). The draft pre-fill result #B includes the KV data of the first candidate token #B and the draft token sequence #B0.

[0241] The draft pre-fill result #B in step S1040b above is the same as the draft pre-fill result #B in step S740 above. For details not elaborated here, please refer to the relevant description in step S740 above.

[0242] Optionally, the draft model can also cache the KV data of the draft token sequence #B0 into the target cache space #B, so that the draft model can reuse it in the decoding stage corresponding to the prompt word #B in subsequent drafts.

[0243] It should be understood that the working principle of the draft model performing the above step S1040b is the same as the working principle of the draft model performing the above steps S1010b to S1030b, and will not be described in detail here.

[0244] For example, taking an initial draft lexical sequence B that includes 3 target lexical blocks B as an example, Figure 11 The diagram shows the time period during which the target model prefills the third target word block B corresponding to the prompt word #B, and the time period during which the draft model prefills the first draft word block B corresponding to the prompt word #B. These two time periods overlap, with a time period #2. It should be understood that... Figure 11 The data preprocessing shown in section 2 is used to convert the prompt word #B into an initial draft lexical sequence B.

[0245] It should be understood that the above Figure 10 The data processing methods provided are for illustrative purposes only and do not constitute any limitation on the data processing methods provided in the embodiments of this application.

[0246] In this embodiment, the pre-filling stage of the draft model corresponding to each prompt word is a block pre-filling stage, and the pre-filling stage of the target model corresponding to each prompt word is also a block pre-filling stage. The draft model located on the GPU performs the pre-filling stage of one draft word block (e.g., the w1st draft word block A) corresponding to each prompt word (e.g., prompt word #A), which is executed in parallel with the target model located on the NPU performing the pre-filling stage of one target word block (e.g., the w2nd target word block A) corresponding to each prompt word. Based on the above settings, the pre-filling stage latency of the draft model can be saved, the model inference can be accelerated, and the data processing efficiency can be improved.

[0247] Example 3

[0248] For example, Figure 12 This is a schematic diagram of a data processing method 1200 provided in an embodiment of this application. The method 1200 can be performed by... Figure 2 The terminal 200A and server 200B shown are... Figure 3 The mobile phone shown or Figure 4 The illustrated electronic device 100 performs this action. See also... Figure 12 The method 1200 includes steps S1210 and S1220, which are described below.

[0249] In step S1210, the first input text is subjected to a first speculative sampling process to generate a first output text. The first speculative sampling process includes N rounds of first iterations. The i-th round of first iteration includes running a draft model to generate the i-th round of candidate words based on the i-th round of first draft data and running a target model to verify the i-th round of candidate words and generate the i-th round of first iteration results. When i equals 1, the i-th round of first draft data includes the first word sequence corresponding to the first input text. When i is an integer greater than 1 and less than or equal to N, the i-th round of first draft data includes the first word sequence and the (i-1) rounds of first iteration results corresponding to the previous (i-1) rounds of first iterations. The first output text is obtained by concatenating the N rounds of first iteration results corresponding to the N rounds of first iterations, where N is a positive integer.

[0250] In step S1220, the acquired second input text undergoes a second speculative sampling process to generate a second output text. This second speculative sampling process includes M rounds of second iterations. The j-th round of second iterations includes a second draft decoding operation that runs a draft model to generate candidate words based on the j-th round of second draft data, and a second target decoding operation that runs a target model to verify the candidate words and generate the results of the j-th round of second iterations. When j equals 1, the j-th round of second draft data includes the second word sequence corresponding to the second input text; when j is an integer greater than 1 and less than or equal to M... When counting, the second draft data of the j-th round includes the second word sequence and the second iteration results of the (j-1) rounds corresponding to the previous (j-1) rounds of the second iteration. The second output text is obtained by concatenating the second iteration results of the M rounds corresponding to the second iteration, where M is a positive integer. Among them, the processing time of the first iteration of the N rounds and the processing time of the second iteration of the M rounds have at least one first overlap time. The at least one first overlap time includes the overlap time between the processing time of at least one round of the first target decoding operation in the first iteration of the N rounds and the processing time of at least one round of the second draft decoding operation in the second iteration of the M rounds.

[0251] The data processing method provided in this application embodiment can be applied to an electronic device that has a first processor deployed with a target model and a second processor deployed with a draft model, such as... Figure 4 The illustrated electronic device 100 includes a first processor 110A and a second processor 110B. In other words, in this embodiment, the target model and the draft model are decoupled by deploying them on two separate processors.

[0252] There are no limitations on the first and second processors. For example, the first and second processors can be two different processors. The first processor can be a portion of a single processor, and the second processor can be the remaining portion of that single processor excluding that portion. There are also no limitations on the types of the first and second processors. In one example, the first processor is a Neural Processing Unit (NPU), and the second processor is a Graphics Processing Unit (GPU). In this example, the target model can be run by running the Neural Processing Unit, and the draft model can be run by running the Graphics Processing Unit. Optionally, the first processor can also be a processor other than an NPU, and the second processor can also be a processor other than a GPU.

[0253] The first input text is used to request the execution of a first user task, and the second input text is used to request the execution of a second user task. The first input text may be different from the second input text.

[0254] It should be understood that there are no restrictions on the names of the draft model and the target model. For example, the target model can also be called the first model, and the draft model can also be called the second model, etc. Optionally, the number of parameters in the draft model is less than the number of parameters in the target model.

[0255] The first iteration of round i includes the first draft decoding operation of round i, which involves running the draft model to generate candidate words for round i based on the first draft data of round i, and the first target decoding operation of round i, which involves running the target model to verify the candidate words for round i and generate the result of the first iteration of round i. In other words, the first iteration of round i includes the first draft decoding operation of round i and the first target decoding operation of round i. The first draft decoding operation of round i includes running the draft model to generate candidate words for round i based on the first draft data of round i; the first target decoding operation of round i includes running the target model to verify the candidate words for round i and generate the result of the first iteration of round i. Similarly, the second iteration of round j includes the second draft decoding operation of round j, which involves running the draft model to generate candidate words for round j based on the second draft data of round j, and the second target decoding operation of round j, which involves running the target model to verify the candidate words for round j and generate the result of the second iteration of round j. In other words, the second iteration of round j includes the second draft decoding operation of round j and the second target decoding operation of round j. The second draft decoding operation of round j includes running the draft model to generate candidate words of round j based on the second draft data of round j; the second target decoding operation of round j includes running the target model to verify the candidate words of round j and generate the second iteration result of round j.

[0256] The first overlap time in step S1220 above includes the processing time of at least one round of first target decoding operation in N rounds of first iteration, and the overlap time of the processing time of at least one round of second draft decoding operation in M ​​rounds of second iteration, which means that at least one round of first target decoding operation executed by the electronic device running the target model and at least one round of second draft decoding operation executed by the electronic device running the draft model are executed in parallel.

[0257] The length of the overlap between the processing time of at least one first target decoding operation in the Nth round of the first iteration and the processing time of at least one second draft decoding operation in the Mth round of the second iteration is not limited. For example, the length of this overlap is less than or equal to the processing time of at least one second draft decoding operation.

[0258] There are no restrictions on the number of iterations for at least one first target decoding operation in N rounds of the first iteration, and the number of iterations for at least one second draft decoding operation in M ​​rounds of the first iteration. N and M can be the same or different, and there are no restrictions on this.

[0259] For example, taking N equal to 2 and M equal to 1 as an example, Figure 8 The diagram shows the overlap between the processing time of at least one first overlap time, decode(TM)_1{jobA} (i.e., the first target decoding operation in the first iteration of the first round of N rounds of first iteration), and the processing time of guess(DM)_1{jobB} (i.e., the second draft decoding operation in the first iteration of the first round of M rounds of first iteration).

[0260] Optionally, in step S1220 above, at least one round of first target decoding operation in the N rounds of first iteration is the i-th round of first target decoding operation, and at least one round of second draft decoding operation in the M rounds of second iteration is the i-th round of second draft decoding operation; if the first condition is met, at least one first overlap time also includes the processing time of the (i+1)-th round of first draft decoding operation and the overlap time with the processing time of the i-th round of second target decoding operation.

[0261] In one example, satisfying the first condition includes: i is a positive integer less than or equal to N, M is a positive integer greater than or equal to N, and N rounds of the first iteration include the (i+1)th round of the first draft decoding operation.

[0262] In the above example, that is, when at least one round of the first target decoding operation in the N rounds of the first iteration is the i-th round of the first target decoding operation, and at least one round of the second draft decoding operation in the M rounds of the second iteration is the i-th round of the second draft decoding operation, and when the N rounds of the first iteration include the (i+1)-th round of the first draft decoding operation, at least one first overlap time also includes the processing time of the (i+1)-th round of the first draft decoding operation and the overlap time with the processing time of the i-th round of the second target decoding operation, where i is a positive integer less than or equal to N, and M is a positive integer greater than or equal to N.

[0263] In another example, satisfying the first condition includes: i is a positive integer less than or equal to M, M is a positive integer less than N, and N is a positive integer greater than 1.

[0264] In the above example, that is, when at least one round of the first target decoding operation in the N rounds of the first iteration is the i-th round of the first target decoding operation, and at least one round of the second draft decoding operation in the M rounds of the second iteration is the i-th round of the second draft decoding operation, and when i is a positive integer less than or equal to M, and M is a positive integer less than N, at least one first overlap time also includes the overlap time between the processing time of the (i+1)-th round of the first draft decoding operation and the processing time of the i-th round of the second target decoding operation, where N is a positive integer greater than 1.

[0265] Optionally, the first speculative sampling process in step S1210 above may further include a first pre-filling process performed before the first iteration of N rounds, wherein the first pre-filling process includes a first target pre-filling result and a first draft pre-filling result, the first lexical sequence includes Q first target lexical blocks, where Q is a positive integer greater than 1; and the first target pre-filling process includes running a target model to generate a k-th first target block result based on the k-th first target lexical block, to generate the first target pre-filling result; where k is a positive integer less than or equal to Q; the first draft pre-filling process includes running a draft model to generate a k-th first draft block result based on the k-th first draft lexical block, to generate the first draft pre-filling result; the k-th first draft lexical block is determined based on the corresponding k-th first target lexical block, and the first draft data of the i-th round includes the first lexical sequence, the first target pre-filling result, and the first draft pre-filling result.

[0266] The first lexical sequence includes Q first target lexical blocks, where Q is a positive integer greater than 1, meaning that the lexical sequence obtained after lexical transformation of the first input text is divided into multiple first target lexical blocks.

[0267] The first target pre-filling process is the procedure of running the target model to pre-fill the first word sequence. Optionally, the first target pre-filling process may also include a procedure for data preprocessing of the first input text, which is a procedure preceding the procedure for pre-filling the first word sequence. The procedure for data preprocessing of the first input text is used to run the target model to perform format conversion processing on the first input text to obtain the first word sequence. For example, see... Figure 13 Before the target model pre-fills the first word sequence [A0, A1, A2, ..., AQ], the target model first executes the data preprocessing flow of the first input text, where A1 represents the first first target word block, A2 represents the second first target word block, and so on, and AQ represents the Qth first target word block; A1' represents the first first draft word block, A2' represents the second first draft word block, and so on, and AQ' represents the Qth first draft word block.

[0268] The first draft pre-filling process is the process of running the draft model to pre-fill the k first draft word blocks corresponding to the first word sequence.

[0269] The first target pre-filling result may include Q first target word blocks corresponding to Q first target segmentation results, and the k-th first target segmentation result may include the key-value data of each target word in the k-th first target word block. Optionally, the target model may also cache the key-value data of each target word in the k-th first target word block for reuse in subsequent target model decoding stages (i.e., the N rounds of first target decoding operations included in the N rounds of the first iteration). The first draft pre-filling result may include Q first draft word blocks corresponding to Q first draft pre-filling results, and the k-th first draft segmentation result may include the key-value data of each draft word in the k-th first draft word block. Optionally, the draft model may also cache the key-value data of each target word in the k-th first draft word block for reuse in subsequent draft model decoding stages (i.e., the N rounds of first draft decoding operations included in the N rounds of the first iteration).

[0270] There are no limitations on the first target pre-filling result and the first draft pre-filling result. The first target pre-filling result may include the target word at the first position in the first output word sequence corresponding to the first output text. Optionally, the first target pre-filling result may also include the key-value data of the first word sequence. The first draft pre-filling result may include the first candidate word. Optionally, the first draft pre-filling result may also include the key-value data of the first word sequence and the key-value data of the target word at the first position in the first output word sequence.

[0271] The k-th first draft word block is determined based on the corresponding k-th first target word block without limitation. In one example, the k-th first draft word block is the k-th first target word block. In another example, when k is greater than or equal to 1 and less than Q, the k-th first draft word block can be determined based on the remaining target words in the k-th first target word block excluding the first target word, and the first target word in the (k+1)-th first target word block. Specifically, the k-th first draft word block is obtained by concatenating the first target word in the (k+1)-th first target word block with the remaining target words in the k-th first target word block excluding the first target word.

[0272] The implementation methods for the first target pre-filling process and the first draft pre-filling process in the above examples are not limited.

[0273] In one example, the processing time for the target model to generate the w-th first target block result overlaps with the processing time for the draft model to generate the (w-1)-th first draft block result, where w is a positive integer greater than 1 and less than or equal to Q.

[0274] The processing time for the target model to generate the w-th first target block result overlaps with the processing time for the draft model to generate the (w-1)-th first draft block result. In other words, the target model generating the w-th first target block result and the draft model generating the (w-1)-th first draft block result are executed in parallel.

[0275] In another example, the processing time of the first target pre-filling process is earlier than the processing time of the first draft pre-filling process. Optionally, in this example, the value of Q can also be equal to 1, that is, the first word sequence is not segmented.

[0276] The processing time of the first target pre-filling process is earlier than that of the first draft pre-filling process, meaning that the first target pre-filling process and the first draft pre-filling process are executed sequentially.

[0277] In the example above, after the target model processes the first word sequence to generate the first target pre-filled result, the draft model processes the first word sequence and the first target pre-filled result to generate the first draft pre-filled result.

[0278] For example, taking Q = 2 as an example, the above-mentioned first target pre-filling process can be Figure 6 The prefill(TM){jobA} shown above can be the first draft prefill process. Figure 6 The example shown is prefill(DM){jobA}. Figure 6 The two processes shown are executed sequentially.

[0279] Optionally, the second speculative sampling process in step S1220 above may further include a second pre-filling process performed before the second iteration of round M, wherein the second pre-filling process includes a second target pre-filling result and a second draft pre-filling result, the second lexical sequence includes Q2 second target lexical blocks, where Q2 is a positive integer greater than 1; and the second target pre-filling process includes running the target model to generate the k2th second target block result based on the k2th second target lexical block, to generate the second target pre-filling result; wherein k2 is a positive integer less than or equal to Q2; the second draft pre-filling process includes running the draft model based on the k2th second draft... The word block generates the k2th second draft block result to generate the second draft pre-fill result; the k2th second draft word block is determined based on the corresponding k2th second target word block; the i-th round of second draft data includes the second word sequence, the second target pre-fill result, and the second draft pre-fill result; among them, the processing time of the target model generating the w2th second target block result based on the w2th second target word block overlaps with the processing time of the draft model generating the (w2-1)th second draft block result based on the (w2-1)th second draft word block, where w2 is a positive integer greater than 1 and less than or equal to Q2.

[0280] The value of Q2 in the second pre-filling process described above may be the same as or different from the value of Q in the first pre-filling process described above, and there is no limitation on this.

[0281] Optionally, the second target pre-filling process may further include a data preprocessing procedure for the second input text, which precedes the pre-filling procedure for the second word sequence. The data preprocessing procedure for the second input text is used to run the target model to perform format conversion processing on the second input text, obtaining the second word sequence. For example, see... Figure 13 Before the target model pre-fills the second word sequence [B0, B1, B2, ..., BQ2], the target model first performs a data preprocessing procedure for the second input text, where B1 represents the first second target word block, B2 represents the second second target word block, and so on, with BQ2 representing the Qth second target word block; B1' represents the first second draft word block, B2' represents the second second draft word block, and so on, with BQ2' representing the Qth second draft word block.

[0282] It should be understood that the working principle of the second pre-filling process is the same as that of the first pre-filling process. For details not elaborated here, please refer to the description above.

[0283] Optionally, the above method further includes: performing a third speculative sampling process on the acquired third input text to generate a third output text, wherein the third speculative sampling process includes L rounds of third iterations, the r-th round of third iterations includes running a draft model to generate r-th round candidate lexical units based on the r-th round of third draft data, and running a target model to verify the r-th round candidate lexical units to generate the r-th round of third iteration results, and running a target model to verify the r-th round candidate lexical units and generate the r-th round of third iteration results; when r equals 1, the r-th round of third draft data includes the third lexical unit sequence corresponding to the third input text; when r is an integer greater than 1 and less than or equal to L, the r-th round of third draft data includes the third lexical unit sequence and the (r-1)-th round of third iteration results corresponding to the previous (r-1) rounds of third iterations, and the third output text is the L-th round of third iteration results concatenated with the L-th round of third iterations. After processing, L is a positive integer; where, after the second iteration of round M is completed, the first H rounds of the first iteration of round N are completed, and the remaining (NH) rounds of the first iteration of round N (excluding the first H rounds of the first iteration) have not been executed, the processing time of the third iteration of round L has at least one second overlap time with the processing time of the remaining (NH) rounds of the first iteration. The at least one second overlap time includes the overlap time between the processing time of at least one round of the first target decoding operation in the remaining (NH) rounds of the first iteration and the processing time of at least one round of the third draft decoding operation; the at least one first overlap time includes the overlap time between the processing time of at least one round of the first target decoding operation in the first H rounds of the first iteration and the processing time of at least one round of the second draft decoding operation in the second iteration of round M, where H is a positive integer less than N.

[0284] The third input text is used to request the execution of a third user task. For example, the first, second, and third input texts can be three different input texts to execute three different user tasks.

[0285] The overlap between the processing time of at least one round of the first target decoding operation in the remaining (NH) rounds of the first iteration and the processing time of at least one round of the third draft decoding operation means that the execution of at least one round of the first target decoding operation corresponding to the first input text by the target model located in the first processor and the execution of at least one round of the third draft decoding operation corresponding to the third input text by the draft model located in the second processor are executed in parallel. There are no restrictions on the iteration rounds corresponding to the at least one round of the first target decoding operation in the remaining (NH) rounds of the first iteration, nor on the iteration rounds corresponding to the at least one round of the third draft decoding operation.

[0286] Optionally, in one example, if at least one first target decoding operation in the remaining (NH) rounds of the first iteration is the t-th round of the first target decoding operation, and at least one third draft decoding operation in the L-th round of the third iteration is the z-th round of the third draft decoding operation, and if the remaining (NH) rounds of the first iteration include the (t+1)-th round of the first draft decoding operation, the at least one second overlap time in the above example also includes the processing time of the (t+1)-th round of the first draft decoding operation and the overlap time with the processing time of the z-th round of the second target decoding operation, where t is a positive integer less than or equal to (NH), and z is a positive integer less than or equal to L.

[0287] The first iteration of round t is any one of the first iterations of the remaining (NH) rounds, where the first iteration of round t includes the first target decoding operation and the first draft decoding operation of round t. t is a positive integer less than or equal to (NH), i.e., t = 1, ..., (NH). There are no restrictions on the relationship between t and z; for example, t can be equal to z, or t can be greater than or less than z.

[0288] In the above example, at least one second overlap time includes the overlap time between the processing time of the first target decoding operation in round t and the processing time of the third draft decoding operation in round z. In the case that the remaining (NH) rounds of the first iteration include the first draft decoding operation in round (t+1), at least one second overlap time also includes the overlap time between the processing time of the first draft decoding operation in round (t+1) and the processing time of the second target decoding operation in round z.

[0289] For example, taking N = 4, M = 1, H = 2, and L = 1 as an example, see [link to example]. Figure 14 At least one first overlap time includes overlap time 1 and overlap time 2, wherein overlap time 1 is the overlap time between the processing time of decode(TM)_1{jobA} (i.e., the first target decoding operation in the first iteration of the first round, i=1) and the processing time of guess(DM)_1{jobB} (the second draft decoding operation in the second iteration of the first round, i=1); overlap time 2 is the overlap time between the processing time of guess(DM)_2{jobA} (i.e., the first draft decoding operation in the first iteration of the second round, i+1=2) and the processing time of decode(TM)_1{jobB} (i.e., the second target decoding operation in the second iteration of the first round, i=1). See also Figure 14In the processing time after overlap time 2, that is, when the second iteration corresponding to the second input text has been completed and the remaining first iterations (including decode(TM)_2{jobA}, guess(DM)_3{jobA}, decode(TM)_3{jobA}, guess(DM)_4{jobA}, decode(TM)_4{jobA}) corresponding to the first input text have not yet been executed, and when the electronic device needs to perform a third speculative sampling process on the new third input text (the process is shown by the dashed rounded rectangle), see [link to relevant documentation]. Figure 14 The content within the dashed rectangle shown includes at least one second overlap time, comprising overlap time 3 and overlap time 4. Overlap time 3 is the overlap time between the processing time of decode(TM)_3{jobA} (i.e., the first target decoding operation in the first iteration of the third round, t=3) and the processing time of guess(DM)_1{jobC} (i.e., the third draft decoding operation in the first round, z=1). Overlap time 4 is the overlap time between the processing time of guess(DM)_4{jobA} (i.e., the first draft decoding operation in the first iteration of the fourth round, t+1=4) and the processing time of decode(TM)_1{jobC} (i.e., the third target decoding operation in the third iteration of the first round, z=1).

[0290] In the example above, when all decoding operations corresponding to the second input text (i.e., the M rounds of the second iteration process) have been completed, some decoding operations corresponding to the first input text (i.e., the first H rounds of the first iteration process) have been completed, and the remaining decoding operations corresponding to the first input text (i.e., the remaining (NH) rounds of the first iteration) have not yet been executed, and the electronic device needs to perform speculative sampling on the new third input text, by setting the target model located on the first processor to execute at least one round of the first target decoding operation in the remaining (NH) rounds of the first iteration corresponding to the first input text, and the draft model located on the second processor to execute at least one round of the third draft decoding operation corresponding to the third input text, the execution is parallel. This ensures the full operation of the first and second processors, avoids the phenomenon of the electronic device serially executing a single task (e.g., the remaining (NH) rounds of the first iteration process corresponding to the first input text), reduces the total data processing latency, and further improves data processing efficiency.

[0291] Optionally, the third speculative sampling process in the above example further includes a third target pre-filling process and a third draft pre-filling process performed before the third iteration of L rounds, wherein the third lexical sequence includes P third target lexical blocks, where P is a positive integer greater than 1; and the third target pre-filling process includes running the target model to generate the h-th third target pre-filling result based on the h-th third target lexical block, to generate the third target pre-filling result; wherein h is a positive integer less than or equal to P; the third draft block pre-filling process includes running the draft model to generate the h-th third target pre-filling result based on the h-th third target lexical block. The draft lexical block generates the h-th third draft pre-fill result to generate the third draft pre-fill result; the h-th third draft lexical block is determined based on the corresponding h-th third target lexical block; the j-th round of third draft data includes the third lexical sequence, the third target pre-fill result, and the third draft pre-fill result; among them, the processing time for running the target model to generate the v-th third target pre-fill result overlaps with the processing time for running the draft model to generate the (v-1)-th third draft pre-fill result, where v is a positive integer greater than 1 and less than or equal to P.

[0292] For example, continue with Figure 14 For example, Figure 14 The processing time of prefill(TM){jobC} (i.e., the third target prefill process) shown within the dashed rectangle overlaps with the processing time of prefill(DM){jobC} (i.e., the third draft prefill process).

[0293] The above example illustrates the working principle of performing third pre-filling processing (including third target pre-filling processing and third draft pre-filling processing) on ​​third input text. This is the same as the working principle of performing first pre-filling processing (including first target pre-filling processing and first draft pre-filling processing) on ​​first input text as described above. For details not elaborated here, please refer to the relevant descriptions above.

[0294] It should be noted that the above Figure 12 For details not elaborated in the provided data processing methods, please refer to the above text. Figure 7 or Figure 10 The relevant descriptions of the provided data processing methods will not be repeated here.

[0295] It should be understood that the above Figure 12The data processing methods shown are merely illustrative and do not constitute any limitation on the data processing methods provided in the embodiments of this application. For example, in step S1220 above, the overlapping time of at least one first overlap time, including the processing time of at least one round of first target decoding operation in N rounds of first iteration, and the overlapping time of at least one round of second draft decoding operation in M ​​rounds of second iteration, is used as an example to introduce the inventive concept of parallel execution of the decoding operation of the target model of one task and the decoding operation of the draft model of another task, provided in the embodiments of this application. Optionally, the phrase "at least one first overlap time, including the overlapping time of the processing time of at least one round of first target decoding operation in N rounds of first iteration, and the overlapping time of the processing time of at least one round of second draft decoding operation in M ​​rounds of second iteration" in step S1220 above can also be replaced with "at least one first overlap time, including the overlapping time of the processing time of at least one round of first draft decoding operation in N rounds of first iteration, and the overlapping time of the processing time of at least one round of second target decoding operation in M ​​rounds of second iteration".

[0296] This application also provides a computer program product that, when executed by a processor, implements the data processing method described in any of the method embodiments of this application.

[0297] The computer program product can be stored in memory, for example, it is a program. The program is eventually converted into an executable object file that can be executed by the processor after processes such as preprocessing, compilation, assembly and linking.

[0298] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the data processing method described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0299] This application also provides a chip for use in an electronic device. The chip includes one or more processors that invoke computer instructions to cause the electronic device to execute the data processing method described in any of the method embodiments of this application.

[0300] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0301] It should be understood that "at least one" as used in this application refers to one or more, and "more than one" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can represent A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0302] Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0303] It should be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0304] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0305] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0306] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0307] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0308] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0309] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method, applied to an electronic device having a first processor with a target model deployed and a second processor with a draft model deployed, comprises: The first input text is subjected to a first speculative sampling process to generate a first output text. The first speculative sampling process includes N rounds of first iterations. The i-th round of first iteration includes running the draft model to generate an i-th round of first draft decoding operation based on the i-th round of first draft data, and running the target model to verify the i-th round of candidate words to generate an i-th round of first iteration results. When i equals 1, the i-th round of first draft data includes the first word sequence corresponding to the first input text. When i is an integer greater than 1 and less than or equal to N, the i-th round of first draft data includes the first word sequence and the (i-1)-th round of first iteration results corresponding to the previous (i-1) rounds of first iterations. The first output text is obtained by concatenating the N-th round of first iteration results corresponding to the N-th round of first iterations, where N is a positive integer. The acquired second input text is subjected to a second speculative sampling process to generate a second output text. The second speculative sampling process includes M rounds of second iterations. The j-th round of second iterations includes running the draft model to generate j-th round candidate lexical units based on the j-th round of second draft data, and running the target model to verify the j-th round candidate lexical units and generate the j-th round of second iteration results, which is a j-th round of second target decoding. When j equals 1, the j-th round of second draft data includes the second lexical unit sequence corresponding to the second input text. When j is an integer greater than 1 and less than or equal to M, the j-th round of second draft data includes the second lexical unit sequence and the (j-1)-th round of second iteration results corresponding to the previous (j-1) rounds of second iterations. The second output text is obtained by concatenating the M-th round of second iteration results corresponding to the M-th round of second iterations, where M is a positive integer. Wherein, the processing time of the Nth round of the first iteration and the processing time of the Mth round of the second iteration have at least one first overlap time, the at least one first overlap time includes the overlap time between the processing time of at least one round of the first target decoding operation in the Nth round of the first iteration and the processing time of at least one round of the second draft decoding operation in the Mth round of the second iteration.

2. The method according to claim 1, characterized in that, At least one first target decoding operation in the N rounds of the first iteration is the i-th round of the first target decoding operation, and at least one second draft decoding operation in the M rounds of the second iteration is the i-th round of the second draft decoding operation; If the first condition is met, the at least one first overlap time also includes the processing time of the (i+1)th round of first draft decoding operation and the overlap time between the processing time of the i-th round of second target decoding operation; Wherein, satisfying the first condition includes: i is a positive integer less than or equal to N, M is a positive integer greater than or equal to N, and the Nth round of first iteration includes the (i+1)th round of first draft decoding operation; or, The condition of satisfying the first condition includes: i is a positive integer less than or equal to M, M is a positive integer less than N, and N is a positive integer greater than 1.

3. The method according to claim 1 or 2, characterized in that, The first speculative sampling process further includes a first target pre-filling process and a first draft pre-filling process performed before the first iteration of the N rounds, wherein the first lexical sequence includes Q first target lexical blocks, where Q is a positive integer greater than 1; and, The first target pre-filling process includes running the target model to generate the kth first target segmentation result based on the kth first target word block, so as to generate the first target pre-filling result; where k is a positive integer less than or equal to Q; The first draft pre-filling process includes running the draft model to generate the kth first draft block result based on the kth first draft word block, so as to generate the first draft pre-filling result; the kth first draft word block is determined based on the corresponding kth first target word block, and the i-th round of first draft data includes the first word sequence, the first target pre-filling result and the first draft pre-filling result; The processing time for running the target model to generate the w-th first target block result based on the w-th first target word block overlaps with the processing time for running the draft model to generate the (w-1)-th first draft block result based on the (w-1)-th first draft word block, where w is a positive integer greater than 1 and less than or equal to Q.

4. The method according to claim 1 or 2, characterized in that, The method further includes: The acquired third input text is subjected to a third speculative sampling process to generate a third output text. This third speculative sampling process includes L rounds of third iterations. The r-th round of third iterations includes running the draft model to generate r-th round candidate lexical units based on the r-th round of draft data, and running the target model to verify the r-th round candidate lexical units and generate the r-th round of third iteration results, resulting in a third-round target decoding operation. When r equals 1, the r-th round of draft data includes the third lexical unit sequence corresponding to the third input text. When r is an integer greater than 1 and less than or equal to L, the r-th round of draft data includes the third lexical unit sequence and the (r-1)-th round of third iteration results corresponding to the previous (r-1) rounds of third iterations. The third output text is obtained by concatenating the L-th round of third iteration results corresponding to the L-th round of third iterations, where L is a positive integer. Wherein, after the completion of the M rounds of second iteration, the completion of the first H rounds of first iteration in the N rounds of first iteration, and the remaining (NH) rounds of first iteration excluding the first H rounds of first iteration in the N rounds of first iteration, the processing time of the L rounds of third iteration has at least one second overlap time with the processing time of the remaining (NH) rounds of first iteration. The at least one second overlap time includes the overlap time between the processing time of at least one round of first target decoding operation in the remaining (NH) rounds of first iteration and the processing time of at least one round of third draft decoding operation. The at least one first overlap time includes the overlap time between the processing time of at least one round of first target decoding operation in the first H rounds of first iteration and the processing time of at least one round of second draft decoding operation in the M rounds of second iteration, where H is a positive integer less than N.

5. The method according to claim 4, characterized in that, The third speculative sampling process further includes a third target pre-filling process and a third draft pre-filling process performed before the third iteration of the L rounds, wherein the third lexical sequence comprises P third target lexical blocks, where P is a positive integer greater than 1; and, The third target pre-filling process includes running the target model to generate the h-th third target pre-filling result based on the h-th third target word block, so as to generate the third target pre-filling result; where h is a positive integer less than or equal to P; The third draft pre-filling process includes running the draft model to generate the h-th third draft pre-filling result based on the h-th third draft lexical block, thereby generating the third draft pre-filling result; the h-th third draft lexical block is determined based on the corresponding h-th third target lexical block, and the j-th round of third draft data includes the third lexical sequence, the third target pre-filling result, and the third draft pre-filling result; The processing time for running the target model to generate the vth third target pre-fill result overlaps with the processing time for running the draft model to generate the (v-1)th third draft pre-fill result, where v is a positive integer greater than 1 and less than or equal to P.

6. The method according to claim 1 or 2, characterized in that, The first processor is a neural network processor, and the second processor is a graphics processor.

7. The method according to claim 1 or 2, characterized in that, The number of parameters in the draft model is less than the number of parameters in the target model.

8. An electronic device, characterized in that, It includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program, which, when executed by the one or more processors, causes the electronic device to perform the data processing method as described in any one of claims 1 to 7.

9. A chip system applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is used to invoke computer instructions to cause the electronic device to perform the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the data processing method as described in any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1 to 7.

Images