Large model distributed inference acceleration method and device based on speculative sampling

Abstract

The embodiment of the present application provides a large-model distributed reasoning acceleration method and device based on speculative sampling, which relates to the field of artificial intelligence. The terminal prefix information is input into the draft small model for content generation to obtain multiple candidate tokens and corresponding small-model probability distributions, and the parameter sequence, each candidate token and the small-model probability value corresponding to the candidate token are transmitted to the base station, so that the base station uses the target large model to generate the large-model probability distribution of each candidate token, and divides the candidate tokens into acceptance tokens and / or rejection tokens. If the terminal receives the large-model probability distribution corresponding to the first rejection token, the first rejection token is resampled to obtain a resampled token, the resampled token is written into the parameter sequence, and the terminal prefix information is updated according to the resampled token, and it is iteratively executed until the inference result is obtained. The rejection / acceptance process of the candidate token is executed on the base station, and the resampling process after rejection is executed on the terminal device to improve the efficiency of collaborative reasoning.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a large-model distributed reasoning acceleration method and device based on speculative sampling. Background Art

[0002] Autoregressive generation is the core mechanism used by large language models to generate text. It operates on an iterative sequence prediction process: the model first receives an initial prefix or context as input, computes a probability distribution over the vocabulary based on the current input sequence, and selects the next token using a sampling strategy. The newly generated token is appended to the end of the input sequence, forming a new context that serves as input for the next prediction. This process repeats until the model outputs an end-of-line (EOS) symbol or the preset maximum generation length is reached.

[0003] Related technologies enable distributed deployment through collaborative inference of heterogeneous edge and end models to improve inference speed. A lightweight model with fewer parameters is deployed on the end to generate candidate tokens. A high-performance model with more parameters is deployed on the edge to verify and correct the candidate tokens, and the final results are returned to the end to update the input prefix. In each interaction between the end and the edge, the end's uplink transmission load is excessive, increasing communication latency and affecting overall inference acceleration efficiency. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a large-model distributed reasoning acceleration method and device based on speculative sampling to improve the distributed deployment reasoning efficiency of large language models.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a large model distributed inference acceleration method, which is applied to a terminal device, on which a small draft model is deployed. The method includes:

[0006] Input the terminal prefix information into the draft small model to generate content, and obtain multiple candidate tokens and a small model probability distribution corresponding to each candidate token;

[0007] Transmitting the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal generated data to a base station, so that the base station uses a target large model to generate a large model probability distribution for each candidate token according to the terminal generated data, and classifies the candidate tokens into accepted tokens and / or rejected tokens, wherein the parameter amount of the target large model is higher than that of the draft small model;

[0008] If the large model probability distribution corresponding to the first rejection token is received, resampling the first rejection token to obtain a resampled token, writing the resampled token into the parameter sequence, and updating the terminal prefix information according to the resampled token;

[0009] Iterate and execute until the end condition is triggered and the inference result is obtained.

[0010] In some embodiments, if the large model probability distribution corresponding to the first rejection token is received, resampling the first rejection token to obtain a resampled token includes:

[0011] Obtain the small model probability distribution corresponding to the first rejection token, and calculate the difference between the corresponding large model probability distribution and the small model probability distribution;

[0012] The difference is non-negatively normalized to obtain a resampling probability distribution, and the resampling token is selected from the vocabulary based on the resampling probability distribution.

[0013] In some embodiments, the iterative execution continues until an end condition is triggered and an inference result is obtained, and the method further includes:

[0014] If a base station token generated by the base station according to the candidate token is received, the parameter sequence is initialized, and the terminal prefix information is updated according to the multiple candidate tokens and the base station token.

[0015] To achieve the above objectives, a second aspect of an embodiment of the present application proposes a large model distributed reasoning acceleration method, which is applied to a base station, where a target large model is deployed. The method includes:

[0016] Receiving terminal-generated data from a terminal device, obtaining a current parameter sequence, each candidate token, and a probability value of a small model corresponding to the candidate token from the terminal-generated data, wherein the candidate token is generated by a draft small model in the terminal device according to terminal prefix information, and the target large model has a higher number of parameters than the draft small model;

[0017] Obtaining base station prefix information based on the parameter sequence, and inputting the base station prefix information into the target large model for content generation, thereby obtaining a base station token and a large model probability distribution corresponding to each candidate token;

[0018] Classifying the candidate tokens into acceptance tokens and / or rejection tokens according to the corresponding large model probability distribution and the small model probability value, obtaining base station generated data according to the large model probability distribution or the base station token corresponding to the first rejection token, and sending the base station generated data to the terminal device;

[0019] Iterate and execute until the end condition is triggered to obtain the complete reasoning result.

[0020] In some embodiments, obtaining updated prefix information based on the parameter sequence includes:

[0021] Obtaining base station prefix information corresponding to a previous iteration and a plurality of acceptance tokens, where an initial value of the base station prefix information is consistent with an initial value of the terminal prefix information;

[0022] The plurality of acceptance tokens and the parameter sequences are sequentially concatenated after the base station prefix information to update the base station prefix information.

[0023] In some embodiments, classifying the candidate tokens into accepted tokens according to the corresponding large model probability distribution and the small model probability value includes:

[0024] Obtaining a large model probability value corresponding to each candidate token from the large model probability distribution;

[0025] For the candidate token, if the corresponding small model probability value is less than or equal to the large model probability value, the candidate token is used as the accepted token.

[0026] In some embodiments, classifying the candidate token as a rejected token according to the corresponding large model probability distribution and the small model probability value includes:

[0027] Obtain the large model probability value corresponding to each candidate token from the large model probability distribution;

[0028] For the candidate token, if the corresponding small model probability value is greater than the large model probability value, a random probability value is generated within a preset range. If the random probability value is greater than or equal to the quotient of the large model probability value and the small model probability value, the candidate token is used as the rejection token; otherwise, the candidate token is used as the acceptance token.

[0029] To achieve the above objectives, a third aspect of the embodiments of the present application proposes a large-model distributed reasoning acceleration device, which is applied to a terminal device, on which a small draft model is deployed. The device includes:

[0030] Candidate content generation module: used to input terminal prefix information into the draft small model to generate content, and obtain multiple candidate tokens and the probability distribution of the small model corresponding to each candidate token;

[0031] A parameter sending module is configured to transmit the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal generated data to a base station, so that the base station generates a large model probability distribution for each candidate token based on the terminal generated data using a target large model, and classifies the candidate tokens into acceptance tokens and / or rejection tokens, wherein the parameter amount of the target large model is higher than that of the draft small model;

[0032] A resampling module is configured to, if receiving the large model probability distribution corresponding to the first rejection token, resample the first rejection token to obtain a resampled token, write the resampled token into the parameter sequence, and update the terminal prefix information according to the resampled token;

[0033] Terminal iteration module: used for iterative execution until the end condition is triggered and the inference result is obtained.

[0034] To achieve the above objectives, a fourth aspect of the embodiments of the present application proposes a large-model distributed inference acceleration device, which is applied to a base station, on which a target large model is deployed, and the device includes:

[0035] A data receiving module is configured to receive terminal-generated data from a terminal device, and obtain a current parameter sequence, each candidate token, and a probability value of a small model corresponding to the candidate token from the terminal-generated data. The candidate token is generated by a draft small model in the terminal device based on terminal prefix information, and the target large model has a higher number of parameters than the draft small model.

[0036] A large model parsing module is configured to obtain base station prefix information based on the parameter sequence, input the base station prefix information into the target large model for content generation, and obtain a base station token and a large model probability distribution corresponding to each candidate token;

[0037] An acceptance / rejection module is configured to classify the candidate tokens into acceptance tokens and / or rejection tokens according to the corresponding large model probability distribution and the small model probability value, obtain base station generated data according to the large model probability distribution or the base station token corresponding to the first rejection token, and send the base station generated data to the terminal device;

[0038] Base station iteration module: used for iterative execution until the end condition is triggered and the inference result is obtained.

[0039] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first or second aspect above when executing the computer program.

[0040] To achieve the above-mentioned purpose, the sixth aspect of the embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method described in the first or second aspect above.

[0041] The embodiment of the present application proposes a large-model distributed reasoning acceleration method and device based on speculative sampling, which inputs the terminal prefix information into the draft small model for content generation, obtains multiple candidate tokens and the small model probability distribution corresponding to each candidate token, and transmits the current parameter sequence, each candidate token and the small model probability value of the candidate token in the corresponding small model probability distribution as the terminal generation data to the base station, so that the base station uses the target large model to generate the large model probability distribution of each candidate token according to the terminal generation data, and divides the candidate tokens into acceptance tokens and / or rejection tokens. If the large model probability distribution corresponding to the first rejection token is received, the first rejection token is resampled to obtain a resampled token, the resampled token is written into the parameter sequence, and the terminal prefix information is updated according to the resampled token, and the execution is iterative until the end condition is triggered to obtain a complete reasoning result. The embodiment of the present application further decomposes the verification process of the candidate token, places the rejection / acceptance process of the candidate token on the base station for execution, and places the resampling process after rejection on the terminal device for execution, so that the terminal device does not need to upload the small model probability distribution corresponding to each candidate token, but only needs to upload the corresponding small model probability value to participate in the rejection / acceptance judgment process. The base station only needs to return the large model probability distribution of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, and return the resampling result to the base station for the next iterative process. In this way, the uplink transmission of multiple small model probability distributions is converted into the downlink transmission of a single large model probability distribution, and the amount of uplink transmission data from the terminal device to the base station is significantly reduced. Even if the downlink data volume of the base station increases slightly, the overall data volume is reduced, and the efficiency of collaborative reasoning is still improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1This is a flowchart of a large-model distributed reasoning acceleration method applied to terminal devices, provided in an embodiment of the present application.

[0043] Figure 2 This is an overall schematic diagram of the large-model distributed reasoning acceleration method provided in an embodiment of the present application.

[0044] Figure 3 This is a flowchart provided by an embodiment of the present application, which shows that if a large model probability distribution corresponding to a first rejection token is received, the first rejection token is resampled to obtain a resampled token.

[0045] Figure 4 This is an optional flowchart of a large-model distributed reasoning acceleration method applied to a base station provided in an embodiment of the present application.

[0046] Figure 5 This is a flowchart of obtaining updated prefix information based on a parameter sequence provided by an embodiment of the present application.

[0047] Figure 6 This is a flowchart provided by an embodiment of the present application for classifying candidate tokens into accepted tokens based on the corresponding large model probability distribution and small model probability values.

[0048] Figure 7 This is a flowchart provided by an embodiment of the present application for classifying candidate tokens into rejected tokens based on the corresponding large model probability distribution and small model probability values.

[0049] Figure 8 This is a flowchart of the large-model distributed reasoning acceleration method provided in an embodiment of the present application.

[0050] Figure 9 This is a structural block diagram of a large-model distributed inference acceleration device applied to a terminal device provided by another embodiment of the present application.

[0051] Figure 10 This is a structural block diagram of a large-model distributed inference acceleration device applied to a base station provided by another embodiment of the present application.

[0052] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flowchart.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0056] First, let’s analyze some of the terms used in this application:

[0057] Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0058] Autoregressive generation is the core mechanism used by large language models to generate text. It operates on an iterative sequence prediction process: the model first receives an initial prefix or context as input, computes a probability distribution over the vocabulary based on the current input sequence, and selects the next token using a sampling strategy. The newly generated token is appended to the end of the input sequence, forming a new context that serves as input for the next prediction. This process repeats until the model outputs an end-of-line (EOS) symbol or the preset maximum generation length is reached.

[0059] Related technologies enable distributed deployment through collaborative inference of heterogeneous edge and end models to improve inference speed. A lightweight model with fewer parameters is deployed on the end to generate candidate tokens. A high-performance model with more parameters is deployed on the edge to verify and correct the candidate tokens, and the final results are returned to the end to update the input prefix. In each interaction between the end and the edge, the end's uplink transmission load is excessive, increasing communication latency and affecting overall inference acceleration efficiency.

[0060] Based on this, the embodiment of the present application provides a large-model distributed reasoning acceleration method and device based on speculative sampling, which further decomposes the verification process of the candidate token, executes the rejection / acceptance process of the candidate token on the base station, and executes the resampling process after rejection on the terminal device. Therefore, the terminal device does not need to upload the small model probability distribution corresponding to each candidate token, but only needs to upload the corresponding small model probability value to participate in the rejection / acceptance judgment process. The base station only needs to return the large model probability distribution of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, and return the resampling result to the base station for the next iterative process. In this way, the amount of uplink transmission data from the terminal device to the base station is significantly reduced. Even if the amount of downlink data of the base station increases slightly, the overall amount of data is reduced, and the efficiency of collaborative reasoning is still improved.

[0061] The embodiments of the present application provide a large-model distributed reasoning acceleration method and device based on speculative sampling, which are specifically illustrated by the following embodiments. First, the large-model distributed reasoning acceleration method in the embodiments of the present application is described.

[0062] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, enabling them to have the capabilities of perception, reasoning, and decision-making.

[0063] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0064] The large-model distributed reasoning acceleration method provided in the embodiment of the present application relates to the field of artificial intelligence. The large-model distributed reasoning acceleration method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be a computer program running in a terminal or a server side. For example, the computer program can be a native program or software module in the operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, such as a client that supports large-model distributed reasoning acceleration, that is, a program that only needs to be downloaded to a browser environment to run; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form. Among them, the terminal communicates with the server through a network. The large-model distributed reasoning acceleration method can be executed by the terminal or the server, or by the terminal and the server in collaboration.

[0065] In some embodiments, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, or smartwatch. Furthermore, the terminal can also be an intelligent in-vehicle device. This intelligent in-vehicle device applies the large-model distributed inference acceleration method of this embodiment to provide related services and enhance the driving experience. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. It can also be a service node in a blockchain system, where each service node in the blockchain system forms a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP). The terminal and server can be connected via Bluetooth, Universal Serial Bus (USB), or a network connection, etc., which is not limited in this embodiment.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0067] The following describes the large model distributed reasoning acceleration method in an embodiment of the present application.

[0068] Figure 1 This is an optional flowchart of the large model distributed reasoning acceleration method provided in the embodiment of the present application. Figure 1 The method is applied to the terminal device (end side), which may include but is not limited to steps 110 to 140. It can also be understood that this embodiment is for Figure 1 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0069] Step 110: Input the terminal prefix information into the draft small model to generate content, and obtain multiple candidate tokens and a small model probability distribution of each candidate token.

[0070] In one embodiment, a lightweight large language model with few parameters is deployed on the terminal device as a draft small model , input the terminal prefix information Prefix into the draft model for content generation, and continuously select candidate tokens, denoted as , and at the same time, we get the vocabulary probability distribution corresponding to each candidate token, which is called the small model probability distribution. That is to say, we get Small model probability distribution .

[0071] In natural language processing (NLP) and language models, a vocabulary is the set of all legal language units (tokens) that a model can understand and generate. It forms the foundation for the model's text processing and determines which words, symbols, or subwords it can recognize. A model can only generate tokens that exist in the vocabulary. Therefore, a larger vocabulary allows the model to process a richer vocabulary, but also increases computational costs. A small vocabulary can lead to unrecognized words. A token is the smallest unit of text processing and can include: full words (such as "cat", "running"), subwords (such as "un" + "happy" = "unhappy"), punctuation (such as "",", "!"), and special control characters (such as [CLS] and [SEP] used in models like BERT).

[0072] In one embodiment, referring to Figure 2 , Figure 2 This is a schematic diagram of the overall large model distributed reasoning acceleration method provided by the embodiment of this application. Assume that a small draft model is deployed on the terminal device. , deploy the target large model on the base station , the number of parameters of the target large model is greater than that of the draft small model, and both are large language models. Assuming that the prefix information of the target large model is called the target prefix information, and the prefix information of the draft small model is called the terminal prefix information, then the initial values ​​of the terminal prefix information and the target prefix information are consistent, that is, both are generated based on the same starting point. For example, in the first iteration, the terminal device can send the acquired terminal prefix information to the base station as the initial value of the target prefix information. In the subsequent iterations, based on the content generation process of the initial value, the base station can perform the target prefix information update through the data acquired by itself, without the terminal device sending the terminal prefix information in the subsequent iterations.

[0073] For example, if Prefix="The black cat" is used as the terminal prefix information, "The black cat" is also used as the target prefix information. First, the terminal device generates the content of "The black cat" using the draft model, continuously generating γ = 4 candidate tokens through autoregression, and recording the probability distribution of the model at each generation step.

[0074] Assume that the probability distribution of the small models corresponding to the four candidate tokens are expressed as follows:

[0075] {"jumps":0.55,"is":0.25,"jumps":0.10,"meows":0.05,... (other tokens have lower probabilities)}, then "jumps" with the highest probability is selected as the first candidate token, that is, ;

[0076] {"over":0.50,"quietly":0.20,"down":0.15,"still":0.10,...}, then select "over" with the highest probability as the second candidate token, that is, ;

[0077] {"the":0.60,"a":0.25,"its":0.10,...}, then select "the" with the highest probability as the third candidate token, that is, ;

[0078] {"table":0.65,"grass":0.25,"road":0.10,...}, then select "table" with the highest probability as the fourth candidate token, that is, .

[0079] Similarly, multiple candidate tokens and corresponding small model probability distributions are generated one by one according to the set number of candidate tokens.

[0080] Step 120: The current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution are transmitted to the base station as terminal generated data.

[0081] In one embodiment, the parameter sequence is updated with each iteration. The specific update process will be described in the subsequent steps. In the first iteration, the parameter sequence y is initialized to a null value sequence, that is, At this time, first determine the small model probability value corresponding to each candidate token , the small model probability value is the value of the corresponding candidate token in the corresponding small model probability distribution. Figure 2 For example, the probability value of the small model corresponding to the first candidate token is The probability value of the small model corresponding to the second candidate token is 0.55, the probability value of the small model corresponding to the third candidate token is 0.6, and the probability value of the small model corresponding to the fourth candidate token is 0.65.

[0082] At this time, the terminal device only needs to set the parameter sequence corresponding to the current iteration process , each candidate token and the small model probability value of the candidate token in the corresponding small model probability distribution can be used as the terminal generated data to be transmitted to the base station, and there is no need to transmit the small model probability distribution corresponding to each candidate token.

[0083] This process will be described in detail in the following embodiment on the base station side.

[0084] In one embodiment, after the base station receives the terminal generated data, it analyzes whether the candidate token is qualified. In this process, the target large model is used to generate the large model probability distribution of each candidate token. The calculation process for the large model probability distribution is identical to that for the small model, with the same vocabulary. The only difference lies in the model used. With the large model probability distribution, candidate tokens can be classified as accepted and / or rejected. If a candidate token passes, it is retained as an accepted token; if it fails, it is rejected as a rejected token. It is understood that if a candidate token is rejected, all subsequent candidate tokens do not need to be further evaluated and are all rejected.

[0085] Step 130: If the large model probability distribution corresponding to the first rejection token is received, the first rejection token is resampled to obtain a resampled token, the resampled token is written into the parameter sequence, and the terminal prefix information is updated according to the resampled token.

[0086] In one embodiment, if the base station partially accepts the candidate token, it needs to return the serial number of the first rejected token and the large model probability distribution of the rejected token to the terminal device. Figure 2 , assuming that the base station determines that all candidate tokens starting from the third are rejected tokens, then the index of the third candidate token (the first rejected token) is returned as 3, and the probability distribution of the third large model is returned , where the large model probability distribution The probability of including each word in the vocabulary as a token in the third position.

[0087] In one embodiment, if the terminal device receives the large model probability distribution corresponding to the first rejected token, it means that the base station has not received all the candidate tokens included in the terminal generated data. At this time, the index information is used to determine which candidate token needs to be resampled. Figure 3 , Figure 3 This is a flowchart of an embodiment of the present application for resampling the first rejection token to obtain a resampled token if a large model probability distribution corresponding to the first rejection token is received, specifically including the following steps:

[0088] Step 310: Obtain the small model probability distribution corresponding to the first rejected token, and calculate the difference between the corresponding large model probability distribution and the small model probability distribution.

[0089] In one embodiment, assuming that the first rejection token is the n+1th candidate token, it is represented as , at this time the probability distribution of the small model of the candidate token is , the probability distribution of the large model is , the difference is expressed as:

[0090]

[0091] Step 320: Perform non-negative normalization on the difference to obtain a resampling probability distribution, and select a resampling token from the vocabulary based on the resampling probability distribution.

[0092] In one embodiment, the resampled probability distribution Expressed as:

[0093]

[0094] in, To ensure that the value is non-negative, Used to normalize numerical values.

[0095] After having the resampling probability distribution, we can select a word from the vocabulary as the resampling token according to the resampling probability distribution, which is expressed as .

[0096] Next, write the resampling token into the parameter sequence At this time, all candidate tokens and resampled tokens received by the base station constitute a sequence , splice the sequence after the terminal prefix information to update the terminal prefix information. The updated terminal prefix information is expressed as Prefix=Prefix+ .

[0097] In one embodiment, referring to Figure 2 , assuming that the third candidate token is the first rejection token returned by the base station, the probability distribution of the large model returned by the base station is and Index=3. After the terminal device receives it, the probability distribution based on the small model is And the large model probability distribution is Compute the resampled probability distribution , and resample from the resampled probability distribution to obtain the resampled token, assuming , write it into the parameter sequence , the candidate token and resampled token received by the base station constitute a sequence , update the terminal prefix information according to Prefix=Prefix+ , get Prefix = "The black cat jumps over a".

[0098] In one embodiment, if the base station receives all the candidate tokens sent to it, the base station will Candidate tokens generate the The token is used as the base station token, and then the base station token is sent to the terminal device. If the terminal device receives the base station token, the sequence number of the token is Index= , expressed as , the candidate token and base station token form a sequence , splice the sequence after the terminal prefix information to update the terminal prefix information. The updated terminal prefix information is expressed as Prefix=Prefix+ .

[0099] Step 140: Iterate and execute until the end condition is triggered to obtain the inference result.

[0100] In one embodiment, after obtaining the updated terminal prefix information, the process is iterated again according to step 110 until a termination condition is triggered and a complete inference result is obtained. The termination condition may be the model output end symbol (EOS) or reaching a preset maximum generation length, which is not limited in this embodiment.

[0101] In the embodiment of the present application, the terminal prefix information is input into the draft small model for content generation, and multiple candidate tokens and the small model probability distribution corresponding to each candidate token are obtained. The current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution are transmitted to the base station as the terminal generation data, so that the base station uses the target large model to generate the large model probability distribution of each candidate token according to the terminal generation data, and divides the candidate tokens into acceptance tokens and / or rejection tokens. If the large model probability distribution corresponding to the first rejection token is received, the first rejection token is resampled to obtain a resampled token, the resampled token is written into the parameter sequence, and the terminal prefix information is updated according to the resampled token. The execution is iterative until the end condition is triggered to obtain the inference result.

[0102] The embodiment of the present application further decomposes the verification process of the candidate token, places the rejection / acceptance process of the candidate token on the base station, and places the resampling process after rejection on the terminal device. Thus, the terminal device does not need to upload the small model probability distribution corresponding to each candidate token, but only needs to upload the corresponding small model probability value to participate in the rejection / acceptance judgment process. The base station only needs to return the large model probability distribution of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, and return the resampling result to the base station for the next iterative process. In this way, the amount of uplink transmission data from the terminal device to the base station is significantly reduced. Even if the amount of downlink data of the base station increases slightly, the overall amount of data is reduced, and the efficiency of collaborative reasoning is still improved.

[0103] Next, the execution process of the large-model distributed reasoning acceleration method provided in the embodiment of the present application in the base station is described.

[0104] In one embodiment, Figure 4 This is an optional flowchart of the large model distributed reasoning acceleration method provided in the embodiment of the present application. Figure 4 The method is applied to the base station (edge ​​side), which may include but is not limited to steps 410 to 440. It can also be understood that this embodiment is for Figure 4 The order of step 410 to step 440 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0105] Step 410: Receive terminal generated data from the terminal device, and obtain the current parameter sequence, each candidate token, and the small model probability value corresponding to the candidate token from the terminal generated data.

[0106] In one embodiment, as described above, the candidate token is generated by the draft small model in the terminal device according to the terminal prefix information, and the parameter amount of the target large model is higher than that of the draft small model.

[0107] Step 420: Obtain base station prefix information based on the parameter sequence, and input the base station prefix information into the target large model for content generation to obtain the base station token and the large model probability distribution corresponding to each candidate token.

[0108] In one embodiment, referring to Figure 5 , Figure 5 This is a flowchart of obtaining updated prefix information based on a parameter sequence provided by an embodiment of the present application, which specifically includes the following steps:

[0109] Step 510: Obtain the base station prefix information and multiple acceptance tokens corresponding to the previous iteration.

[0110] In one embodiment, the initial value of the base station prefix information is consistent with the initial value of the terminal prefix information. That is, in the first iteration, the base station prefix information and the terminal prefix information are consistent. In subsequent iterations, the base station prefix information is updated based on the current parameter sequence in the terminal-generated data. At this time, the base station prefix information and multiple accepted tokens from the previous iteration need to be obtained.

[0111] by Figure 2 For example, in the second iteration, the previous iteration is the first iteration, so the base station prefix information corresponding to the previous iteration is "The black cat". At this time, the base station accepts as well as , therefore, the accepted token is 、 , and the parameter sequence at this time is .

[0112] Step 520: Multiple acceptance tokens and parameter sequences are sequentially concatenated after the base station prefix information, and the base station prefix information is updated.

[0113] In one embodiment, multiple acceptance tokens and parameter sequences are sequentially spliced ​​after the base station prefix information, and the base station prefix information is updated. Figure 2 For example, the updated base station prefix information is: "The black cat jumps over a". It can be seen that after the update, the base station prefix information and the terminal prefix information in the same round are consistent.

[0114] In each iteration, the obtained base station prefix information is input into the target large model, and content is generated based on the received multiple candidate tokens to obtain the base station token and the large model probability distribution corresponding to each candidate token. .

[0115] Step 430: Classify the candidate tokens into acceptance tokens and / or rejection tokens according to the corresponding large model probability distribution and small model probability values, obtain the base station generated data according to the large model probability distribution or base station token corresponding to the first rejection token, and send the base station generated data to the terminal device.

[0116] In one embodiment, each candidate token needs to be accepted / rejected based on its corresponding large model probability distribution and small model probability value. Figure 6 , Figure 6 This is a flowchart of classifying candidate tokens into accepted tokens based on the corresponding large model probability distribution and small model probability value provided by an embodiment of the present application, specifically including the following steps:

[0117] Step 610: Obtain the large model probability value corresponding to each candidate token from the large model probability distribution.

[0118] In one embodiment, since the large model probability distribution is also generated based on the vocabulary, it contains the probability value corresponding to the candidate token, which is called the large model probability value and is expressed as .

[0119] Step 620: For a candidate token, if the corresponding small model probability value is less than or equal to the large model probability value, the candidate token is taken as an accepted token.

[0120] In one embodiment, for the i-th candidate token, it is represented as , the corresponding small model probability value is , if the probability value of the small model is less than or equal to the probability value of the large model, that is, , it is considered that the prediction of the draft small model is not "overconfident" and is consistent with or more conservative than the prediction of the target large model. Therefore, no correction is required. The generation of the candidate token meets the expectations of the target large model, so the candidate token can be directly used as the acceptance token, thereby maintaining generation efficiency and avoiding unnecessary correction calculations.

[0121] In one embodiment, referring to Figure 7 , Figure 7 This is a flowchart provided by an embodiment of the present application for classifying candidate tokens into rejected tokens based on the corresponding large model probability distribution and small model probability value, which specifically includes the following steps:

[0122] Step 710: Obtain the large model probability value corresponding to each candidate token from the large model probability distribution.

[0123] Step 720: For a candidate token, if the corresponding small model probability value is greater than the large model probability value, a random probability value is generated within a preset range. If the random probability value is greater than or equal to the quotient of the large model probability value and the small model probability value, the candidate token is treated as a rejection token; otherwise, the candidate token is treated as an acceptance token.

[0124] In one embodiment, for the i-th candidate token, it is represented as , the corresponding small model probability value is , if the probability value of the small model is greater than the probability value of the large model, that is, , it is considered that the prediction of the draft small model is different from the prediction of the target large model, and it selects a token that the target large model does not recognize, so it needs to be corrected. This is because Think for the draft model The probability that it should occur, The target large model is in the same context The true probability estimate of , indicating that the draft model overestimates If the target large model uses it as a receiving token, the generated result will deviate from the true distribution of the target large model, which may produce low-quality text.

[0125] At this time, the quotient of the large model probability value and the small model probability value is calculated, which is expressed as: , generates a random number in the preset range of [0,1] ,if , then the candidate token is regarded as a rejection token, otherwise the candidate token is regarded as an acceptance token. The probability of accepting the candidate token.

[0126] In one embodiment, after receiving a rejection token, the base station sends the large model probability distribution corresponding to the first rejection token to the terminal device for resampling. If there is no rejection token, the base station only sends the base station token back to the terminal device via the downlink channel to update the terminal prefix information.

[0127] Step 440: Iterate and execute until the end condition is triggered to obtain the inference result.

[0128] In one embodiment, when the terminal-generated data is received again, the above process is iterated until the end condition is triggered and a complete inference result is obtained. The iterative process here corresponds to the iterative process of the terminal device.

[0129] If the verification of candidate token is placed in the base station as in the related art, in addition to the terminal device uploading the generated Candidate tokens In addition, you must upload Small model probability distribution .

[0130] Assumption: The time it takes for the terminal device’s draft model to generate each candidate token is ; The time it takes for the base station's target large model to generate each token is ; The communication time between the terminal device and the base station is Therefore, the time required to complete a complete reasoning process is:

[0131]

[0132] in, Uplink transmission time and downlink transmission time Composition, expressed as:

[0133]

[0134] The uplink transmission time is expressed as:

[0135]

[0136] in, is the uplink transmission rate, Indicates the amount of data to be uploaded.

[0137] For the probability distribution of the uploaded small model, the amount of data to be uploaded is proportional to the size of the vocabulary:

[0138]

[0139] in, is the vocabulary size, for full precision and half precision, The values ​​of are 32bit and 16bit respectively. For example, , =16bit, then Approximately 500kbit. During each interaction between the terminal device and the base station, the amount of uplink data is , so the uplink transmission load is too heavy, resulting in a long transmission time.

[0140] Therefore, the embodiment of the present application further decomposes the verification process of the candidate token, places the rejection / acceptance process of the candidate token on the base station, and places the resampling process after rejection on the terminal device. As a result, the terminal device does not need to upload the small model probability distribution corresponding to each candidate token, but only uploads the corresponding small model probability value to participate in the rejection / acceptance judgment process. The base station only needs to return the large model probability distribution of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, which greatly reduces the amount of data transmitted during interaction and solves the problem of excessive load during uplink transmission.

[0141] In the embodiment of the present application, the uplink transmission time is negligible, and its main communication volume is the large model probability distribution of the first rejected candidate token in the downlink transmission. If all generated candidate tokens are accepted, there is no need to transmit any probability distribution in the downlink.

[0142] Therefore, the average time required for a single inference becomes:

[0143]

[0144]

[0145]

[0146] in, is the average travel time, is the downlink transmission rate, since ,so , that is, the downlink transmission rate is higher, and the average amount of data transmitted per interaction is reduced to the previous , the amount of data is greatly reduced, among which, if If all candidate tokens are accepted, then the downlink transmission only needs to transmit the base station token, and does not need to transmit the large model probability distribution. In this case, the downlink transmission time can be considered to be approximately zero. If at least one token is rejected, the downlink transmission needs to transmit the large model probability distribution. The probability of all tokens being accepted is set to , the probability of partial acceptance is .

[0147] It's understandable that the time required to calculate token probabilities and indices is negligible compared to the vocabulary's probability distribution. For example, the LLaMA-2 vocabulary is 32,000, and the dimensionality of the probability distribution is far greater than the number of generated candidate tokens. During the transmission process, only the transmission of probability distributions significantly impacts inference efficiency. Therefore, this article primarily considers the transmission of vocabulary probability distributions, ignoring other information such as tokens.

[0148] In one embodiment, referring to Figure 8 , Figure 8 This is a flow chart of the large model distributed reasoning acceleration method provided by the embodiment of the present application. Assume that the terminal prefix information is , after the draft model generates content, we get There are 4 candidate tokens, and the corresponding small model probability distribution of the 4 candidate tokens is expressed as: , the small model probability value corresponding to each candidate token is expressed as: At this time, the terminal device transmits the current parameter sequence, each candidate token and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal generation data to the base station. The base station uses the target large model to generate the large model probability distribution of each candidate token. , after having the large model probability distribution, we can classify the candidate tokens into acceptance tokens and / or rejection tokens. If they are qualified, they are retained as acceptance tokens; if they are unqualified, they are rejected as rejection tokens. Assuming that the base station accepts the first two candidate tokens, the third candidate token is used as the large model probability distribution corresponding to the rejection token. It is sent to the terminal device, and the terminal device resamples accordingly.

[0149] If the base station does a token rejection resampling process, the terminal device needs to send 4 small model probability distributions through the uplink transmission process To the base station, the communication volume at this time is 4 , the required uplink transmission time is: In the resampling process of the terminal device, the uplink transmission is changed to downlink transmission, and the amount of data transmitted downlink is , time is .

[0150] The technical solution provided by the embodiment of the present application generates content by inputting the terminal prefix information into the draft small model to obtain multiple candidate tokens and the small model probability distribution corresponding to each candidate token, and transmits the current parameter sequence, each candidate token and the small model probability value of the candidate token in the corresponding small model probability distribution as the terminal generation data to the base station, so that the base station uses the target large model to generate the large model probability distribution of each candidate token according to the terminal generation data, and divides the candidate tokens into acceptance tokens and / or rejection tokens. If the large model probability distribution corresponding to the first rejection token is received, the first rejection token is resampled to obtain a resampled token, the resampled token is written into the parameter sequence, and the terminal prefix information is updated according to the resampled token, and the execution is iterative until the end condition is triggered to obtain a complete inference result. The embodiment of the present application further decomposes the verification process of the candidate token, places the rejection / acceptance process of the candidate token on the base station, and places the resampling process after rejection on the terminal device. Thus, the terminal device does not need to upload the small model probability distribution corresponding to each candidate token, but only needs to upload the corresponding small model probability value to participate in the rejection / acceptance judgment process. The base station only needs to return the large model probability distribution of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, and return the resampling result to the base station for the next iterative process. In this way, the amount of uplink transmission data from the terminal device to the base station is significantly reduced. Even if the amount of downlink data of the base station increases slightly, the overall amount of data is reduced, and the efficiency of collaborative reasoning is still improved.

[0151] The embodiment of the present application also provides a large model distributed reasoning acceleration device, which is applied to a terminal device and can implement the above large model distributed reasoning acceleration method. Figure 9 , the device comprises:

[0152] Candidate content generation module 910: used to input terminal prefix information into the draft small model to generate content, and obtain multiple candidate tokens and the small model probability distribution corresponding to each candidate token.

[0153] Parameter sending module 920: used to transmit the current parameter sequence, each candidate token and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal generation data to the base station, so that the base station uses the target large model to generate the large model probability distribution of each candidate token according to the terminal generation data, and divides the candidate tokens into acceptance tokens and / or rejection tokens. The parameter amount of the target large model is higher than that of the draft small model.

[0154] Resampling module 930: for resampling the first rejection token if the large model probability distribution corresponding to the first rejection token is received, obtaining a resampled token, writing the resampled token into the parameter sequence, and updating the terminal prefix information according to the resampled token.

[0155] Terminal iteration module 940: used for iterative execution until the end condition is triggered and a complete inference result is obtained.

[0156] The specific implementation of the large-model distributed reasoning acceleration device of this embodiment is basically the same as the specific implementation of the above-mentioned large-model distributed reasoning acceleration method applied to the terminal device, and will not be repeated here.

[0157] The embodiment of the present application also provides a large model distributed reasoning acceleration device, which is applied to a base station and can implement the above large model distributed reasoning acceleration method. Figure 10 , the device comprises:

[0158] Data receiving module 1010: used to receive terminal-generated data from the terminal device, obtain the current parameter sequence, each candidate token and the small model probability value corresponding to the candidate token from the terminal-generated data, the candidate token is the draft small model in the terminal device generated according to the terminal prefix information, and the parameter amount of the target large model is higher than that of the draft small model.

[0159] Large model parsing module 1020: used to obtain base station prefix information based on the parameter sequence, and input the base station prefix information into the target large model for content generation, to obtain the base station token and the large model probability distribution corresponding to each candidate token.

[0160] Acceptance / rejection module 1030: used to classify candidate tokens into acceptance tokens and / or rejection tokens according to the corresponding large model probability distribution and small model probability values, and obtain base station generated data according to the large model probability distribution or base station token corresponding to the first rejection token, and send the base station generated data to the terminal device.

[0161] The base station iteration module 1040 is used for iterative execution until the end condition is triggered and a complete inference result is obtained.

[0162] The specific implementation of the large-model distributed reasoning acceleration device of this embodiment is basically consistent with the specific implementation of the above-mentioned large-model distributed reasoning acceleration method applied to the base station, and will not be repeated here.

[0163] An embodiment of the present application further provides an electronic device, including:

[0164] at least one memory;

[0165] at least one processor;

[0166] at least one program;

[0167] The program is stored in a memory, and the processor executes at least one program to implement the large-model distributed inference acceleration method described above. The electronic device can be any intelligent terminal including a mobile phone, tablet computer, personal digital assistant (PDA), in-vehicle computer, etc.

[0168] See also Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0169] The processor 1101 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0170] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called by the processor 1101 to execute the large model distributed inference acceleration method of the embodiments of this application.

[0171] Input / output interface 1103, used to implement information input and output;

[0172] Communication interface 1104, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0173] Bus 1105 , which transmits information between various components of the device (e.g., processor 1101 , memory 1102 , input / output interface 1103 , and communication interface 1104 );

[0174] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via a bus 1105 .

[0175] An embodiment of the present application also provides a storage medium, which is a storage medium that stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned large-model distributed reasoning acceleration method.

[0176] The memory, as a non-transient storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The embodiment of the present application proposes a large-model distributed reasoning acceleration method and device based on speculative sampling, which inputs the terminal prefix information into the draft small model for content generation, obtains multiple candidate tokens and the small model probability distribution corresponding to each candidate token, and transmits the current parameter sequence, each candidate token and the small model probability value of the candidate token in the corresponding small model probability distribution as the terminal generation data to the base station, so that the base station uses the target large model to generate the large model probability distribution of each candidate token according to the terminal generation data, and divides the candidate tokens into acceptance tokens and / or rejection tokens. If the large model probability distribution corresponding to the first rejection token is received, the first rejection token is resampled to obtain a resampled token, the resampled token is written into the parameter sequence, and the terminal prefix information is updated according to the resampled token, and the execution is iterative until the end condition is triggered to obtain a complete reasoning result. The embodiment of the present application further decomposes the verification process of the candidate token, places the rejection / acceptance process of the candidate token on the base station, and places the resampling process after rejection on the terminal device. Thus, the terminal device does not need to upload the small model probability distribution corresponding to each candidate token, but only needs to upload the corresponding small model probability value to participate in the rejection / acceptance judgment process. The base station only needs to return the large model probability distribution of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, and return the resampling result to the base station for the next iterative process. In this way, the amount of uplink transmission data from the terminal device to the base station is significantly reduced. Even if the amount of downlink data of the base station increases slightly, the overall amount of data is reduced, and the efficiency of collaborative reasoning is still improved.

[0178] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0179] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0180] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0181] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0182] The terms "first," "second," "third," "fourth," and the like (if any) in the specification of the present application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0183] It should be understood that in this application, "at least one (item)" means one or more, and "more" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0185] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0187] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0188] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A large model distributed inference acceleration method based on speculative sampling, characterized in that: Applied to a terminal device, a small draft model is deployed on the terminal device, and the method includes: Input the terminal prefix information into the draft small model to generate content, and obtain multiple candidate tokens and a small model probability distribution corresponding to each candidate token; Transmitting the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal generated data to a base station, so that the base station uses a target large model to generate a large model probability distribution for each candidate token according to the terminal generated data, and classifies the candidate tokens into accepted tokens and / or rejected tokens, wherein the parameter amount of the target large model is higher than that of the draft small model; If the large model probability distribution corresponding to the first rejection token is received, resampling the first rejection token to obtain a resampled token, writing the resampled token into the parameter sequence, and updating the terminal prefix information according to the resampled token; Iterate and execute until the end condition is triggered and the inference result is obtained.

2. The large model distributed inference acceleration method based on speculative sampling according to claim 1 is characterized in that: If the large model probability distribution corresponding to the first rejection token is received, resampling the first rejection token to obtain a resampled token includes: Obtain the small model probability distribution corresponding to the first rejection token, and calculate the difference between the corresponding large model probability distribution and the small model probability distribution; The difference is non-negatively normalized to obtain a resampling probability distribution, and the resampling token is selected from the vocabulary based on the resampling probability distribution.

3. The large model distributed inference acceleration method based on speculative sampling according to claim 1 is characterized in that: The iterative execution continues until an end condition is triggered and an inference result is obtained, and the method further includes: If a base station token generated by the base station according to the candidate token is received, the parameter sequence is initialized, and the terminal prefix information is updated according to the multiple candidate tokens and the base station token.

4. A large model distributed reasoning acceleration method based on speculative sampling, characterized in that: Applied to a base station, where a target large model is deployed, the method includes: Receiving terminal-generated data from a terminal device, obtaining a current parameter sequence, each candidate token, and a probability value of a small model corresponding to the candidate token from the terminal-generated data, wherein the candidate token is generated by a draft small model in the terminal device according to terminal prefix information, and the target large model has a higher number of parameters than the draft small model; Obtaining base station prefix information based on the parameter sequence, and inputting the base station prefix information into the target large model for content generation, thereby obtaining a base station token and a large model probability distribution corresponding to each candidate token; Classifying the candidate tokens into acceptance tokens and / or rejection tokens according to the corresponding large model probability distribution and the small model probability value, obtaining base station generated data according to the large model probability distribution or the base station token corresponding to the first rejection token, and sending the base station generated data to the terminal device; Iterate and execute until the end condition is triggered and the inference result is obtained.

5. The large model distributed inference acceleration method based on speculative sampling according to claim 4 is characterized in that: The obtaining updated prefix information based on the parameter sequence includes: Obtaining base station prefix information corresponding to a previous iteration and a plurality of acceptance tokens, where an initial value of the base station prefix information is consistent with an initial value of the terminal prefix information; The plurality of acceptance tokens and the parameter sequences are sequentially concatenated after the base station prefix information to update the base station prefix information.

6. The large model distributed inference acceleration method based on speculative sampling according to claim 4 is characterized in that: Classifying the candidate tokens into accepted tokens according to the corresponding large model probability distribution and the small model probability value includes: Obtaining a large model probability value corresponding to each candidate token from the large model probability distribution; For the candidate token, if the corresponding small model probability value is less than or equal to the large model probability value, the candidate token is used as the accepted token.

7. The large model distributed inference acceleration method based on speculative sampling according to claim 4 is characterized in that: Classifying the candidate token into a rejection token according to the corresponding large model probability distribution and the small model probability value includes: Obtaining a large model probability value corresponding to each candidate token from the large model probability distribution; For the candidate token, if the corresponding small model probability value is greater than the large model probability value, a random probability value is generated within a preset range. If the random probability value is greater than or equal to the quotient of the large model probability value and the small model probability value, the candidate token is used as the rejection token; otherwise, the candidate token is used as the acceptance token.

8. A large model distributed inference acceleration device based on speculative sampling, characterized in that: Applied to a terminal device, a small draft model is deployed on the terminal device, and the device includes: Candidate content generation module: used to input terminal prefix information into the draft small model to generate content, and obtain multiple candidate tokens and the probability distribution of the small model corresponding to each candidate token; A parameter sending module is configured to transmit the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal generated data to a base station, so that the base station generates a large model probability distribution for each candidate token based on the terminal generated data using a target large model, and classifies the candidate tokens into acceptance tokens and / or rejection tokens, wherein the parameter amount of the target large model is higher than that of the draft small model; A resampling module is configured to, if receiving the large model probability distribution corresponding to the first rejection token, resample the first rejection token to obtain a resampled token, write the resampled token into the parameter sequence, and update the terminal prefix information according to the resampled token; Terminal iteration module: used for iterative execution until the end condition is triggered and the inference result is obtained.

9. A large model distributed inference acceleration device based on speculative sampling, characterized in that: Applied to a base station, a target large model is deployed on the base station, and the device includes: A data receiving module is configured to receive terminal-generated data from a terminal device, and obtain a current parameter sequence, each candidate token, and a probability value of a small model corresponding to the candidate token from the terminal-generated data. The candidate token is generated by a draft small model in the terminal device based on terminal prefix information, and the target large model has a higher number of parameters than the draft small model. A large model parsing module is configured to obtain base station prefix information based on the parameter sequence, input the base station prefix information into the target large model for content generation, and obtain a base station token and a large model probability distribution corresponding to each candidate token; An acceptance / rejection module is configured to classify the candidate tokens into acceptance tokens and / or rejection tokens according to the corresponding large model probability distribution and the small model probability value, obtain base station generated data according to the large model probability distribution or the base station token corresponding to the first rejection token, and send the base station generated data to the terminal device; Base station iteration module: used for iterative execution until the end condition is triggered and the inference result is obtained.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the large-model distributed inference acceleration method based on speculative sampling as described in any one of claims 1 to 7.

11. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the large model distributed reasoning acceleration method based on speculative sampling according to any one of claims 1 to 7 is implemented.

Images