Large model distributed reasoning acceleration method and device based on speculation sampling

Abstract

The embodiment of the invention provides a large model distributed reasoning acceleration method and device based on speculation sampling, and relates to the field of artificial intelligence. And inputting the terminal prefix information into a draft small model for content generation to obtain a plurality of candidate tokens and corresponding small model probability distributions, and transmitting the parameter sequence, each candidate token and a small model probability value corresponding to the candidate token to a base station, so that the base station generates a large model probability distribution of each candidate token by using a target large model, and the candidate tokens are divided into accepted tokens and / or rejected tokens. And if the terminal receives the large model probability distribution corresponding to the first rejection token, resampling the first rejection token to obtain a resampling token, writing the resampling token into the parameter sequence, updating terminal prefix information according to the resampling token, and performing iterative execution until a reasoning result is obtained. The rejection / receiving process of the candidate tokens is executed on the base station, and the resampling process after rejection is executed on the terminal equipment, so that the cooperative reasoning efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and particularly to a large model distributed inference acceleration method and device based on speculative sampling. Background Art

[0002] Autoregressive generation is the core mechanism for large language models to achieve text generation. Its working principle is based on an iterative sequence prediction process: the model first receives an initial prefix or context as input, calculates the probability distribution on the vocabulary according to the current input sequence, and selects the next token through a sampling strategy. The newly generated token is appended to the end of the input sequence to form a new context as the input for the next round of prediction. This process is executed cyclically until the model outputs an end-of-sequence (EOS) or reaches a preset maximum generation length.

[0003] In related technologies, distributed deployment can be achieved through the collaborative inference of edge and terminal heterogeneous models to improve the inference speed. A lightweight model with a smaller number of parameters is deployed on the terminal side to generate candidate tokens; a high-performance model with a larger number of parameters is deployed on the edge side to verify and correct the candidate tokens, and the final result is returned to the terminal side for updating the input prefix. In each interaction between the terminal side and the edge side, the uplink transmission load on the terminal side is too large, the communication delay increases, and the overall inference acceleration efficiency is affected. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a large model distributed inference acceleration method and device based on speculative sampling to improve the distributed deployment inference efficiency of large language models.

[0005] To achieve the above object, the first aspect of the embodiments of this application proposes a large model distributed inference acceleration method, which is applied to a terminal device on which a draft small model is deployed. The method includes: Inputting terminal prefix information into the draft small model for content generation to obtain a plurality of candidate tokens and the small model probability distribution corresponding to each candidate token; Transmitting the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal generated data to a base station, so that the base station uses a target large model to generate the large model probability distribution of each candidate token according to the terminal generated data, and classifies the candidate tokens into accepted tokens and / or rejected tokens. The number of parameters of the target large model is higher than that of the draft small model; If the probability distribution of the large model corresponding to the first rejection token is received, resample the first rejection token to obtain a resampled token, write the resampled token into the parameter sequence, and update the terminal prefix information according to the resampled token; Iteratively execute until an end condition is triggered to obtain an inference result.

[0006] In some embodiments, the step of if the probability distribution of the large model corresponding to the first rejection token is received, resampling the first rejection token to obtain a resampled token includes: Obtain the probability distribution of the small model corresponding to the first rejection token, and calculate the difference between the corresponding probability distribution of the large model and the probability distribution of the small model; Perform non - negative normalization on the difference to obtain a resampling probability distribution, and select the resampled token from the vocabulary based on the resampling probability distribution.

[0007] In some embodiments, before the step of iteratively execute until an end condition is triggered to obtain an inference result, the method further includes: If the base station token generated by the base station according to the candidate token is received, initialize the parameter sequence, and update the terminal prefix information according to the multiple candidate tokens and the base station token.

[0008] To achieve the above object, a second aspect of the embodiments of the present application proposes a large - model distributed inference acceleration method, which is applied to a base station, and a target large model is deployed on the base station. The method includes: Receive terminal - generated data from a terminal device, and obtain the current parameter sequence, each candidate token, and the probability value of the small model corresponding to the candidate token from the terminal - generated data. The candidate token is generated by a draft small model in the terminal device according to the terminal prefix information, and the number of parameters of the target large model is higher than that of the draft small model; Obtain base - station prefix information based on the parameter sequence, and input the base - station prefix information into the target large model for content generation to obtain a base - station token and the probability distribution of the large model corresponding to each candidate token; Classify the candidate tokens into acceptance tokens and / or rejection tokens according to the corresponding probability distribution of the large model and the probability value of the small model, and obtain base - station - generated data according to the probability distribution of the large model corresponding to the first rejection token or the base - station token, and send the base - station - generated data to the terminal device; Iteratively execute until the end condition is triggered to obtain the complete inference result.

[0009] In some embodiments, obtaining the updated prefix information based on the parameter sequence includes: Obtain the base station prefix information corresponding to the previous iteration and the multiple acceptance tokens, and the initial value of the base station prefix information is the same as the initial value of the terminal prefix information; Concatenate the multiple acceptance tokens and the parameter sequence after the base station prefix information in sequence to update the base station prefix information.

[0010] In some embodiments, classifying the candidate tokens into acceptance tokens according to the corresponding large model probability distribution and the small model probability value includes: Obtain the large model probability value corresponding to each candidate token from the large model probability distribution; For the candidate token, if the corresponding small model probability value is less than or equal to the large model probability value, use the candidate token as the acceptance token.

[0011] In some embodiments, classifying the candidate tokens into rejection tokens according to the corresponding large model probability distribution and the small model probability value includes: Obtain the large model probability value corresponding to each candidate token from the large model probability distribution; For the candidate token, if the corresponding small model probability value is greater than the large model probability value, generate a random probability value within a preset range. If the random probability value is greater than or equal to the quotient of the large model probability value and the small model probability value, use the candidate token as the rejection token, otherwise use the candidate token as the acceptance token.

[0012] To achieve the above object, a third aspect of the embodiments of the present application proposes a large model distributed inference acceleration device, which is applied to a terminal device, and a draft small model is deployed on the terminal device. The device includes: Candidate content generation module: used to input the terminal prefix information into the draft small model for content generation to obtain multiple candidate tokens and the small model probability distribution corresponding to each candidate token; Parameter Sending Module: It is used to transmit the current parameter sequence, each of the candidate tokens, and the small model probability value of the candidate token in the probability distribution of the corresponding small model as terminal-generated data to the base station, so that the base station uses the target large model to generate the probability distribution of each candidate token according to the terminal-generated data, and classify the candidate tokens into accepted tokens and / or rejected tokens. The parameter quantity of the target large model is higher than that of the draft small model; Resampling Module: If it receives the probability distribution of the large model corresponding to the first rejected token, it is used to resample the first rejected token to obtain a resampled token, write the resampled token into the parameter sequence, and update the terminal prefix information according to the resampled token; Terminal Iteration Module: It is used to perform iterative execution until an end condition is triggered to obtain an inference result.

[0013] To achieve the above object, a fourth aspect of the embodiments of the present application proposes a large model distributed inference acceleration device, which is applied to a base station on which a target large model is deployed. The device includes: Data Receiving Module: It is used to receive terminal-generated data from a terminal device, and obtain the current parameter sequence, each candidate token, and the small model probability value corresponding to the candidate token from the terminal-generated data. The candidate token is generated by a draft small model in the terminal device according to terminal prefix information. The parameter quantity of the target large model is higher than that of the draft small model; Large Model Parsing Module: It is used to obtain base station prefix information based on the parameter sequence, and input the base station prefix information into the target large model for content generation to obtain base station tokens and the probability distribution of the large model corresponding to each candidate token; Accept / Reject Module: It is used to classify the candidate tokens into accepted tokens and / or rejected tokens according to the corresponding large model probability distribution and the small model probability value, and obtain base station-generated data according to the probability distribution of the large model corresponding to the first rejected token or the base station tokens, and send the base station-generated data to the terminal device; Base Station Iteration Module: It is used to perform iterative execution until an end condition is triggered to obtain an inference result.

[0014] To achieve the above object, a fifth aspect of the embodiments of the present application proposes an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the method described in the first aspect or the second aspect above.

[0015] To achieve the above object, a sixth aspect of the embodiments of the present application proposes a storage medium, which is a storage medium storing a computer program, and when the computer program is executed by a processor, the method described in the first aspect or the second aspect above is implemented.

[0016] The method and device for accelerating large model distributed inference based on speculative sampling proposed in the embodiments of the present application input terminal prefix information into a draft small model for content generation, obtaining multiple candidate tokens and the small model probability distribution corresponding to each candidate token, and using the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal generated data to be transmitted to the base station, so that the base station uses the target large model to generate the large model probability distribution of each candidate token according to the terminal generated data, and classifies the candidate tokens into accepted tokens and / or rejected tokens. If the large model probability distribution corresponding to the first rejected token is received, resample the first rejected token to obtain a resampled token, write the resampled token into the parameter sequence, and update the terminal prefix information according to the resampled token, and iterate until the end condition is triggered to obtain the complete inference result. The embodiments of the present application further decompose the verification process of candidate tokens, place the rejection / reception process of candidate tokens at the base station, and place the resampling process after rejection on the terminal device. Thus, the terminal device does not need to upload the small model probability distribution corresponding to each candidate token, and only needs to upload the corresponding small model probability value to participate in the rejection / reception judgment process. The base station only needs to return the large model probability distribution of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, and return the resampled result to the base station for the next iteration process. In this way, the uplink transmission of multiple small model probability distributions is converted into the downlink transmission of a single large model probability distribution. The uplink transmission data volume from the terminal device to the base station is significantly reduced. Even if the downlink data volume of the base station increases slightly, the overall data volume is reduced, and the collaborative inference efficiency is still improved. Description of the Drawings

[0017] Figure 1 It is a flowchart of the method for accelerating large model distributed inference applied to a terminal device provided by the embodiments of the present application.

[0018] Figure 2 It is an overall schematic diagram of the method for accelerating large model distributed inference provided by the embodiments of the present application.

[0019] Figure 3 It is a flowchart of resampling the first rejected token to obtain a resampled token if the large model probability distribution corresponding to the first rejected token is received provided by the embodiments of the present application.

[0020] Figure 4 It is an optional flowchart of the large model distributed inference acceleration method applied to a base station provided by an embodiment of the present application.

[0021] Figure 5 It is a flowchart of obtaining updated prefix information based on a parameter sequence provided by an embodiment of the present application.

[0022] Figure 6 It is a flowchart of classifying candidate tokens into accepted tokens according to the corresponding large model probability distribution and small model probability values provided by an embodiment of the present application.

[0023] Figure 7 It is a flowchart of classifying candidate tokens into rejected tokens according to the corresponding large model probability distribution and small model probability values provided by an embodiment of the present application.

[0024] Figure 8 It is a flowchart of the large model distributed inference acceleration method provided by an embodiment of the present application.

[0025] Figure 9 It is a structural block diagram of a large model distributed inference acceleration device applied to a terminal device provided by another embodiment of the present application.

[0026] Figure 10 It is a structural block diagram of a large model distributed inference acceleration device applied to a base station provided by another embodiment of the present application.

[0027] Figure 11 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0028] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] It should be noted that although functional module division is performed in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the flowchart.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0031] First, several nouns involved in the present application are analyzed: Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence; Artificial Intelligence is a branch of computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. The research in this field includes robots, speech recognition, image recognition, natural language processing, and expert systems, etc. Artificial Intelligence can simulate the information processes of human consciousness and thinking. It also refers to the theory, method, technology, and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0032] Autoregressive generation is the core mechanism for large language models to achieve text generation. Its working principle is based on an iterative sequence prediction process: The model first receives an initial prefix or context as input, calculates the probability distribution on the vocabulary based on the current input sequence, and selects the next token through a sampling strategy. The newly generated token is appended to the end of the input sequence to form a new context as the input for the next round of prediction. This process is executed cyclically until the model outputs an end-of-sequence (EOS) or reaches a preset maximum generation length.

[0033] In related technologies, distributed deployment can be achieved through collaborative inference of edge and terminal heterogeneous models to improve the inference speed. A lightweight model with a smaller number of parameters is deployed on the terminal side to generate candidate tokens; a high-performance model with a larger number of parameters is deployed on the edge side to verify and correct the candidate tokens, and the final result is returned to the terminal side for updating the input prefix. In each interaction between the terminal side and the edge side, the uplink transmission load on the terminal side is too large, and the communication delay increases, affecting the overall inference acceleration efficiency.

[0034] Based on this, the embodiments of this application provide a large model distributed inference acceleration method and device based on speculative sampling. By further decomposing the verification process of candidate tokens, the rejection / acceptance process of candidate tokens is executed at the base station, and the resampling process after rejection is executed on the terminal device. Thus, the terminal device does not need to upload the probability distribution of the small model corresponding to each candidate token, but only uploads the corresponding probability value of the small model to participate in the rejection / acceptance judgment process. The base station only needs to return the probability distribution of the large model of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, and return the resampling result to the base station for the next iterative process. In this way, the uplink transmission data volume from the terminal device to the base station is significantly reduced. Even if the downlink data volume of the base station increases slightly, the overall data volume decreases, and the collaborative inference efficiency remains high.

[0035] The embodiments of the present application provide a large model distributed inference acceleration method and device based on speculative sampling, which will be specifically described through the following embodiments. First, the large model distributed inference acceleration method in the embodiments of the present application will be described.

[0036] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, Artificial Intelligence (AI) is to use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, and to sense the environment, acquire knowledge, and use knowledge to obtain the best results in theory, methods, technologies, and application systems. In other words, artificial intelligence is a comprehensive technology in computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning, and decision-making.

[0037] Artificial intelligence technology is an interdisciplinary subject, involving a wide range of fields, including both hardware-level technologies and software-level technologies. Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0038] The large model distributed inference acceleration method provided by the embodiments of the present application relates to the field of artificial intelligence. The large model distributed inference acceleration method provided by the embodiments of the present application can be applied to terminals, or to server sides, or can also be a computer program running on terminals or server sides. For example, the computer program can be a native program or software module in an operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, such as a client that supports large model distributed inference acceleration, that is, a program that only needs to be downloaded to the browser environment to run; it can also be a small program that can be embedded into any APP. All in all, the above computer program can be any form of application program, module, or plug-in. Among them, the terminal communicates with the server through the network. The large model distributed inference acceleration method can be executed by the terminal or the server, or jointly executed by the terminal and the server.

[0039] In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc. In addition, the terminal can also be a smart vehicle-mounted device. The smart vehicle-mounted device applies the large model distributed inference acceleration method of this embodiment to provide relevant services and enhance the driving experience. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, and the service nodes in the blockchain system form a Peer To Peer (P2P) network, and the P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP) protocol. The terminal and the server can be connected through communication connection methods such as Bluetooth, Universal Serial Bus (USB), or network, and this embodiment does not limit this here.

[0040] This application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet-type devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, small computers, large computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0041] The large model distributed inference acceleration method in the embodiments of this application is described below.

[0042] Figure 1 It is an optional flowchart of the large model distributed inference acceleration method provided by the embodiments of this application. Figure 1 The method in is applied to a terminal device (on the terminal side) and can include, but is not limited to, steps 110 to 140. At the same time, it can be understood that this embodiment Figure 1The order of steps 110 to 140 is not specifically limited, and the order of steps can be adjusted according to actual needs, or some steps can be reduced or added.

[0043] Step 110: Input the terminal prefix information into the draft small model for content generation to obtain multiple candidate tokens and the small model probability distribution of each candidate token.

[0044] In one embodiment, a lightweight large language model with few parameters is deployed on the terminal device as the draft small model , input the terminal prefix information Prefix into the draft small model for content generation, and continuously select candidate tokens in the vocabulary table by means of autoregressive sampling, denoted as , and at the same time obtain the vocabulary table probability distribution corresponding to each candidate token. This vocabulary table probability distribution is called the small model probability distribution. That is to say, small model probability distributions are obtained .

[0045] Among them, in natural language processing (NLP) and language models, the vocabulary table is a set of all legal language units (tokens) that the model can understand and generate. It is the basis for the model to process text and determines which words, symbols or sub-words the model can recognize. The model can only generate tokens that exist in the vocabulary table. Therefore, the larger the vocabulary table, the richer the vocabulary that the model can process, but the higher the computational cost. If the vocabulary table is too small, out-of-vocabulary problems may occur. And a token is the smallest unit of text processing, which can be: a complete word (such as "cat", "running"), a sub-word (such as "un" + "happy" = "unhappy"), a punctuation mark (such as ",", "!"), a special control character (such as [CLS], [SEP] for models such as BERT).

[0046] In one embodiment, referring to Figure 2 , Figure 2 is the overall schematic diagram of the large model distributed inference acceleration method provided by the embodiment of the present application. Assume that a draft small model is deployed on the terminal device, and a target large model , the number of parameters of the target large model is greater than that of the draft small model, and both are large language models. Assume that the prefix information of the target large model is called target prefix information, and the prefix information of the draft small model is called terminal prefix information. Then the initial values of the terminal prefix information and the target prefix information are the same, that is to say, both are generated based on the same starting point. For example, in the first iteration, the terminal device can send the obtained terminal prefix information to the base station as the initial value of the target prefix information. In the subsequent iteration process, based on the content generation process of this initial value, the base station can update the target prefix information through the data it obtains, without the terminal device sending the terminal prefix information in the subsequent iterations.

[0047] Taking Prefix = “The black cat” as an example of the terminal prefix information, at this time, “The black cat” is also used as the target prefix information. First, the terminal device uses the draft small model to generate content for “The black cat”, continuously generates γ = 4 candidate tokens in an autoregressive manner, and records the probability distribution of the small model at each generation step.

[0048] Suppose the probability distributions of the small model corresponding to the 4 candidate tokens are respectively expressed as: {"jumps":0.55,"is":0.25,"jumps":0.10,"meows":0.05,... (the probabilities of other tokens are lower)}, then the "jumps" with the highest probability is selected as the 1st candidate token, that is ; {"over":0.50,"quietly":0.20,"down":0.15,"still":0.10,...}, then the "over" with the highest probability is selected as the 2nd candidate token, that is ; {"the":0.60,"a":0.25,"its":0.10,...}, then the "the" with the highest probability is selected as the 3rd candidate token, that is ; {"table":0.65,"grass":0.25,"road":0.10,...}, then the "table" with the highest probability is selected as the 4th candidate token, that is .

[0049] And so on, generate multiple candidate tokens and the corresponding probability distributions of the small model one by one according to the set number of candidate tokens.

[0050] Step 120: Transmit the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal-generated data to the base station.

[0051] In one embodiment, the parameter sequence is updated with each iteration process. The specific update process will be described in subsequent steps. In the first iteration, the parameter sequence y is an initialized empty value sequence, that is . At this time, first determine the small model probability value corresponding to each candidate token , and the small model probability value is the value of the corresponding candidate token in the corresponding small model probability distribution. Take Figure 2 as an example. The small model probability value corresponding to the 1st candidate token is 0.55, the small model probability value corresponding to the 2nd candidate token is 0.5, the small model probability value corresponding to the 3rd candidate token is 0.6, and the small model probability value corresponding to the 4th candidate token is 0.65.

[0052] At this time, the terminal device only needs to transmit the corresponding parameter sequence , each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal-generated data to the base station, and there is no need to transmit the small model probability distribution corresponding to each candidate token.

[0053] And this process will be described in detail in the following embodiments on the base station side.

[0054] In one embodiment, when the base station receives the terminal-generated data, after receiving the terminal-generated data, the base station analyzes whether the candidate token is qualified accordingly. In this process, the large model probability distribution of each candidate token is generated using the target large model , and the calculation process of the large model probability distribution is the same as that of the small model probability distribution. The two have the same vocabulary, and the only difference is the model used. After having the large model probability distribution, the candidate tokens can be classified into accepted tokens and / or rejected tokens accordingly. If it is qualified, it is retained and used as an accepted token. If it is unqualified, it is rejected and used as a rejected token. It can be understood that if a certain candidate token is rejected, all subsequent candidate tokens do not need to be judged and are all used as rejected tokens.

[0055] Step 130: If the large model probability distribution corresponding to the first rejected token is received, resample the first rejected token to obtain a resampled token, write the resampled token into the parameter sequence, and update the terminal prefix information according to the resampled token.

[0056] In one embodiment, if the base station partially accepts a candidate token, in this case, it is necessary to return the sequence number identifier of the first rejected token and the probability distribution of the large model of this rejected token to the terminal device. Refer to Figure 2 , assuming that the base station determines that all tokens starting from the 3rd candidate token are rejected tokens. In this case, return the Index = 3 of the third candidate token (the first rejected token), and return the probability distribution of the large model of the 3rd , where the probability distribution of the large model includes the possibility of each word in the vocabulary as the token at the third position.

[0057] In one embodiment, if the terminal device receives the probability distribution of the large model corresponding to the first rejected token, it indicates that the base station does not fully accept the candidate tokens included in the data generated by the terminal. In this case, determine which candidate token needs to be resampled according to the Index information. Refer to Figure 3 , Figure 3 is the flowchart of resampling the first rejected token to obtain the resampled token if the probability distribution of the large model corresponding to the first rejected token is received provided by the embodiment of the present application, which specifically includes the following steps: Step 310: Obtain the probability distribution of the small model corresponding to the first rejected token, and calculate the difference between the corresponding probability distribution of the large model and the probability distribution of the small model.

[0058] In one embodiment, assume that the first rejected token is the (n + 1)th candidate token, denoted as , in this case, the probability distribution of the small model of this candidate token is , the probability distribution of the large model is , and the difference is denoted as:

[0059] Step 320: Perform non - negative normalization on the difference to obtain the resampling probability distribution, and select the resampled token from the vocabulary based on the resampling probability distribution.

[0060] In one embodiment, the resampling probability distribution is denoted as:

[0061] where, is used to ensure non - negative values, is used to normalize the value.

[0062] After obtaining the resampling probability distribution, a word can be selected from the vocabulary according to the resampling probability distribution as the resampling token, denoted as .

[0063] Next, write the resampling token into the parameter sequence . At this time, all the candidate tokens received by the base station and the resampling token form a sequence . Concatenate this sequence after the terminal prefix information to update the terminal prefix information. The updated terminal prefix information is denoted as Prefix = Prefix + .

[0064] In one embodiment, referring to Figure 2 , assume that the 3rd candidate token is the first rejected token returned by the base station. At this time, the probability distribution of the large model returned by the base station is and Index = 3. After the terminal device receives it, based on the probability distribution of the small model being and the probability distribution of the large model being calculate the resampling probability distribution , and resample from the resampling probability distribution to obtain the resampling token. Assume , write it into the parameter sequence . The candidate tokens received by the base station and the resampling token form a sequence . Update the terminal prefix information. According to Prefix = Prefix + , obtain Prefix = "The black cat jumps over a".

[0065] In one embodiment, if the base station fully receives all the candidate tokens sent, at this time, the base station will generate the th token as the base station token according to the candidate tokens, and then send the base station token to the terminal device. If the terminal device receives the base station token, the serial number of this token is Index = , denoted as . The candidate tokens and the base station token form a sequence . Concatenate this sequence after the terminal prefix information to update the terminal prefix information. The updated terminal prefix information is denoted as Prefix = Prefix + .

[0066] Step 140: Iteratively execute until the end condition is triggered to obtain the inference result.

[0067] In one embodiment, after obtaining the updated terminal prefix information, iteration is performed again in the manner of step 110 until an end condition is triggered to obtain a complete inference result. The end condition here may be that the model outputs an end-of-sequence token (EOS) or reaches a preset maximum generation length, and this embodiment does not make a limitation on this.

[0068] In the embodiment of the present application, the terminal prefix information is input into a draft small model for content generation to obtain a plurality of candidate tokens and the small model probability distribution corresponding to each candidate token. The current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution are transmitted to the base station as terminal generation data, so that the base station uses the target large model to generate the large model probability distribution of each candidate token according to the terminal generation data, and classifies the candidate tokens into accepted tokens and / or rejected tokens. If the large model probability distribution corresponding to the first rejected token is received, resampling is performed on the first rejected token to obtain a resampled token, the resampled token is written into the parameter sequence, and the terminal prefix information is updated according to the resampled token, and the iteration is executed until an end condition is triggered to obtain an inference result.

[0069] In the embodiment of the present application, the verification process of the candidate tokens is further decomposed. The rejection / acceptance process of the candidate tokens is executed at the base station, and the resampling process after rejection is executed on the terminal device. Thus, the terminal device does not need to upload the small model probability distribution corresponding to each candidate token, and only needs to upload the corresponding small model probability value to participate in the rejection / acceptance judgment process. The base station only needs to return the large model probability distribution of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, and return the resampled result to the base station for the next iteration process. In this way, the uplink transmission data volume from the terminal device to the base station is significantly reduced. Even if the downlink data volume of the base station increases slightly, the overall data volume is reduced, and the collaborative inference efficiency is still increased.

[0070] Next, the execution process of the large model distributed inference acceleration method provided by the embodiment of the present application at the base station is described.

[0071] In one embodiment, Figure 4 is an optional flowchart of the large model distributed inference acceleration method provided by the embodiment of the present application, Figure 4 The method in can be applied to a base station (edge side) and may include but is not limited to steps 410 to 440. At the same time, it can be understood that this embodiment does not make a specific limitation on Figure 4 the order of steps 410 to 440 in, and the order of steps can be adjusted according to actual needs, or some steps can be reduced or added.

[0072] Step 410: Receive the terminal-generated data from the terminal device, and obtain the current parameter sequence, each candidate token, and the small model probability value corresponding to the candidate token from the terminal-generated data.

[0073] In one embodiment, as described above, the candidate token is generated by the draft small model in the terminal device according to the terminal prefix information, and the number of parameters of the target large model is higher than that of the draft small model.

[0074] Step 420: Obtain the base station prefix information based on the parameter sequence, and input the base station prefix information into the target large model for content generation to obtain the base station token and the large model probability distribution corresponding to each candidate token.

[0075] In one embodiment, referring to Figure 5 , Figure 5 is the flowchart for obtaining the updated prefix information based on the parameter sequence provided by the embodiment of the present application, which specifically includes the following steps: Step 510: Obtain the base station prefix information and multiple accepted tokens corresponding to the previous iteration.

[0076] In one embodiment, the initial value of the base station prefix information is the same as the initial value of the terminal prefix information. That is to say, in the first iteration, the base station prefix information and the terminal prefix information are the same. In subsequent iteration processes, the base station prefix information is updated according to the current parameter sequence in the terminal-generated data. At this time, it is necessary to obtain the base station prefix information and multiple accepted tokens in the previous iteration.

[0077] Taking Figure 2 as an example, in the second iteration, the previous iteration is the first iteration. Therefore, the base station prefix information corresponding to the previous iteration is "The black cat". At this time, the base station has accepted and , so the accepted tokens are 、 , and at this time, the parameter sequence is .

[0078] Step 520: Concatenate the multiple accepted tokens and the parameter sequence after the base station prefix information in sequence to update the base station prefix information.

[0079] In one embodiment, concatenate the multiple accepted tokens and the parameter sequence after the base station prefix information in sequence to update the base station prefix information. Referring to the example in Figure 2 , the updated base station prefix information is: "The black cat jumps overa". It can be seen that after the update, the base station prefix information in the same round is the same as the terminal prefix information.

[0080] In each iteration, the obtained base station prefix information is input into the target large model, and content generation is performed based on the received multiple candidate tokens to obtain base station tokens and the large model probability distribution corresponding to each candidate token. 。

[0081] Step 430: Classify the candidate tokens into accepted tokens and / or rejected tokens according to the corresponding large model probability distribution and small model probability value, and obtain base station generated data according to the large model probability distribution corresponding to the first rejected token or the base station token, and send the base station generated data to the terminal device.

[0082] In one embodiment, for each candidate token, it is necessary to make an acceptance / rejection judgment according to its corresponding large model probability distribution and small model probability value. Refer to Figure 6 , Figure 6 is the flowchart for classifying candidate tokens into accepted tokens according to the corresponding large model probability distribution and small model probability value provided by the embodiment of the present application, which specifically includes the following steps: Step 610: Obtain the large model probability value corresponding to each candidate token from the large model probability distribution.

[0083] In one embodiment, since the large model probability distribution is also generated based on the vocabulary, it contains the probability value corresponding to the candidate token, which is called the large model probability value and is denoted as 。

[0084] Step 620: For a candidate token, if the corresponding small model probability value is less than or equal to the large model probability value, the candidate token is used as an accepted token.

[0085] In one embodiment, for the i-th candidate token, which is denoted as , the corresponding small model probability value is , if the small model probability value is less than or equal to the large model probability value, that is, satisfying , it is considered that the prediction of the draft small model is not "overconfident", is consistent with or more conservative than the prediction of the target large model. Therefore, no correction is required, and the generation of this candidate token meets the expectations of the target large model. Therefore, this candidate token can be directly used as an accepted token, thus maintaining the generation efficiency and avoiding unnecessary correction calculations.

[0086] In one embodiment, refer to Figure 7 , Figure 7 is the flowchart for classifying candidate tokens into rejected tokens according to the corresponding large model probability distribution and small model probability value provided by the embodiment of the present application, which specifically includes the following steps: Step 710: Obtain the large model probability value corresponding to each candidate token from the large model probability distribution.

[0087] Step 720: For a candidate token, if the corresponding small model probability value is greater than the large model probability value, generate a random probability value within a preset range. If the random probability value is greater than or equal to the quotient of the large model probability value and the small model probability value, regard the candidate token as a rejected token; otherwise, regard the candidate token as an accepted token.

[0088] In one embodiment, for the i-th candidate token, which is denoted as , the corresponding small model probability value is . If the small model probability value is greater than the large model probability value, that is, it satisfies , it is considered that the prediction of the draft small model is different from that of the target large model, and it selects a token not recognized by the target large model. Therefore, correction is needed. This is because is the probability that the draft small model thinks should appear, while is the true probability estimate of the target large model for in the same context. If , it means that the draft small model overestimates the possibility of (that is, the draft small model selects this token more "confidently" than the target large model). If the target large model takes it as an accepted token, it will cause the generated result to deviate from the true distribution of the target large model and may produce low-quality text.

[0089] At this time, calculate the quotient of the large model probability value and the small model probability value, which is denoted as: . Generate a random number within the preset range of [0, 1]. If , then regard the candidate token as a rejected token; otherwise, regard the candidate token as an accepted token. That is to say, the base station accepts this candidate token with a probability of .

[0090] In one embodiment, after there is a rejected token, the base station sends the large model probability distribution corresponding to the first rejected token to the terminal device for resampling. If there is no rejected token, the base station only needs to send the base station token back to the terminal device through the downlink channel to update the terminal prefix information.

[0091] Step 440: Iteratively execute until the end condition is triggered to obtain the inference result.

[0092] In one embodiment, when the data generated by the terminal is received again, the above process is iterated until the end condition is triggered to obtain a complete inference result. The iteration process here corresponds to the iteration process of the terminal device.

[0093] If, as in the related art, the verification of candidate tokens is placed at the base station, in addition to the need for the terminal device to upload the generated number of candidate tokens in addition, it is also necessary to upload the probability distribution of the small model .

[0094] Assume that: the time for the draft small model of the terminal device to generate each candidate token is ; the time for the target large model of the base station to generate each token is ; the communication time between the terminal device and the base station is . Therefore, the time required to complete a complete inference process is:

[0095] Among them, consists of the uplink transmission time and the downlink transmission time , expressed as:

[0096] Among them, the uplink transmission time, expressed as:

[0097] Among them, is the uplink transmission rate, represents the amount of data to be uploaded.

[0098] For the probability distribution of the uploaded small model, the amount of data to be uploaded is proportional to the size of the vocabulary:

[0099] Among them, is the size of the vocabulary. For full precision and half precision, takes the values of 32bit and 16bit respectively. For example, , =16bit, then is approximately 500kbit. In each interaction process between the terminal device and the base station, the uplink data volume is , so the uplink transmission load is too large, resulting in too long transmission time.

[0100] Therefore, the embodiments of the present application further decompose the verification process of candidate tokens. The rejection / acceptance process of candidate tokens is performed at the base station, and the resampling process after rejection is performed on the terminal device. As a result, the terminal device does not need to upload the probability distribution of the small model corresponding to each candidate token, but only uploads the corresponding probability value of the small model to participate in the rejection / acceptance judgment process. The base station only needs to return the probability distribution of the large model of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, greatly reducing the amount of data transmitted during interaction and solving the problem of excessive load during the uplink transmission process.

[0101] In the embodiments of the present application, the uplink transmission time can be ignored. Its main traffic is the probability distribution of the large model of the first rejected candidate token in the downlink transmission. If all generated candidate tokens are accepted, no probability distribution needs to be transmitted even in the downlink.

[0102] Therefore, the time required for an average single inference becomes:

[0103]

[0104]

[0105] where is the average transit time, is the downlink transmission rate. Since , so , that is to say, the downlink transmission rate is higher, and the amount of data transmitted per average interaction is reduced to of the previous amount. The amount of data is greatly reduced. Among them, if candidate tokens are all accepted, only the base station token needs to be transmitted in the downlink, and the probability distribution of the large model does not need to be transmitted. At this time, it can be considered that the downlink transmission time is approximately zero. If at least one token is rejected, the probability distribution of the large model needs to be transmitted in the downlink. Let the probability of the case where all are accepted be , and the probability of the case where some are received be .

[0106] It can be understood that, compared with the probability distribution of the vocabulary, the time-consuming required for the probability value and index of the token can be ignored. For example, the vocabulary size of LLaMA-2 is 32,000, and the dimension of the probability distribution is much larger than the number of generated candidate tokens. During the transmission process, only the transmission related to the probability distribution will have a greater impact on the inference efficiency. Therefore, here mainly consider the transmission of the probability distribution of the vocabulary, and other information such as tokens is ignored.

[0107] In one embodiment, referring to Figure 8 , Figure 8 is a flowchart of the large model distributed inference acceleration method provided by an embodiment of the present application. Assume that the terminal prefix information is . After content generation by the draft small model, 4 candidate tokens are obtained, and at the same time, there are also small model probability distributions of the corresponding 4 candidate tokens, expressed as: . The small model probability value corresponding to each candidate token is expressed as: . At this time, the terminal device transmits the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal-generated data to the base station. The base station uses the target large model to generate the large model probability distribution of each candidate token. After having the large model probability distribution, the candidate tokens can be classified into accepted tokens and / or rejected tokens accordingly. If qualified, they are retained as accepted tokens, and if unqualified, they are rejected and used as rejected tokens. Assume that the base station accepts the first two candidate tokens, then the large model probability distribution corresponding to the 3rd candidate token as a rejected token is sent to the terminal device, and at this time, the terminal device performs resampling accordingly.

[0108] If the resampling of the rejected token is performed at the base station, the terminal device needs to send 4 small model probability distributions to the base station through the uplink transmission process. At this time, the traffic is 4 , and the required uplink transmission time is: . In the resampling process at the terminal device, the uplink transmission is changed to downlink transmission, and the downlink transmission data volume is , and the time is .

[0109] The technical solution provided by the embodiments of this application generates content by inputting terminal prefix information into a draft small model, obtaining multiple candidate tokens and the small model probability distribution corresponding to each candidate token, and using the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal-generated data to be transmitted to the base station, so that the base station uses the target large model to generate the large model probability distribution of each candidate token based on the terminal-generated data, and classifies the candidate tokens into accepted tokens and / or rejected tokens. If the large model probability distribution corresponding to the first rejected token is received, resampling is performed on the first rejected token to obtain a resampled token, the resampled token is written into the parameter sequence, and the terminal prefix information is updated according to the resampled token, and the iteration is performed until the end condition is triggered to obtain a complete inference result. The embodiments of this application further decompose the verification process of candidate tokens, place the rejection / acceptance process of candidate tokens at the base station, and place the resampling process after rejection on the terminal device. As a result, the terminal device does not need to upload the small model probability distribution corresponding to each candidate token, and only needs to upload the corresponding small model probability value to participate in the rejection / acceptance judgment process. The base station only needs to return the large model probability distribution of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, and return the resampled result to the base station for the next iteration process. In this way, the uplink transmission data volume from the terminal device to the base station is significantly reduced. Even if the downlink data volume of the base station increases slightly, the overall data volume is reduced, and the collaborative inference efficiency is still high.

[0110] The embodiments of this application also provide a large model distributed inference acceleration device, which is applied to a terminal device and can implement the above large model distributed inference acceleration method. Refer to Figure 9 , and the device includes: A candidate content generation module 910: configured to input terminal prefix information into a draft small model for content generation, obtaining multiple candidate tokens and the small model probability distribution corresponding to each candidate token.

[0111] A parameter sending module 920: configured to use the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal-generated data to be transmitted to the base station, so that the base station uses the target large model to generate the large model probability distribution of each candidate token based on the terminal-generated data, and classifies the candidate tokens into accepted tokens and / or rejected tokens, and the number of parameters of the target large model is higher than that of the draft small model.

[0112] Resampling module 930: If the probability distribution of the large model corresponding to the first rejection token is received, resample the first rejection token to obtain a resampled token, write the resampled token into the parameter sequence, and update the terminal prefix information according to the resampled token.

[0113] Terminal iteration module 940: Used for iterative execution until an end condition is triggered to obtain a complete inference result.

[0114] The specific implementation of the large model distributed inference acceleration device in this embodiment is basically the same as the specific implementation of the above large model distributed inference acceleration method applied to the terminal device, and will not be elaborated here.

[0115] This application embodiment also provides a large model distributed inference acceleration device, which is applied to a base station and can implement the above large model distributed inference acceleration method. Refer to Figure 10 , and this device includes: Data receiving module 1010: Used to receive the terminal generated data from the terminal device, and obtain the current parameter sequence, each candidate token, and the small model probability value corresponding to the candidate token from the terminal generated data. The candidate token is generated by the draft small model in the terminal device according to the terminal prefix information, and the number of parameters of the target large model is higher than that of the draft small model.

[0116] Large model parsing module 1020: Used to obtain the base station prefix information based on the parameter sequence, and input the base station prefix information into the target large model for content generation to obtain the base station token and the probability distribution of the large model corresponding to each candidate token.

[0117] Acceptance / rejection module 1030: Used to classify the candidate tokens into acceptance tokens and / or rejection tokens according to the corresponding large model probability distribution and small model probability value, and obtain the base station generated data according to the large model probability distribution corresponding to the first rejection token or the base station token, and send the base station generated data to the terminal device.

[0118] Base station iteration module 1040: Used for iterative execution until an end condition is triggered to obtain a complete inference result.

[0119] The specific implementation of the large model distributed inference acceleration device in this embodiment is basically the same as the specific implementation of the above large model distributed inference acceleration method applied to the base station, and will not be elaborated here.

[0120] This application embodiment also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in a memory, and a processor executes the at least one program to implement the large model distributed inference acceleration method described above in this application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an in-vehicle computer, etc.

[0121] Please refer to Figure 11 , Figure 11 , which schematically shows the hardware structure of an electronic device according to another embodiment. The electronic device includes: A processor 1101, which can be implemented in the form of a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application; A memory 1102, which can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1102 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1102 and are called by the processor 1101 to execute the large model distributed inference acceleration method of the embodiments of this application; An input / output interface 1103, which is used to implement information input and output; A communication interface 1104, which is used to implement communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.); A bus 1105, which transmits information between various components of the device (such as the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104); Among them, the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104 achieve communication connections with each other inside the device through the bus 1105.

[0122] The embodiments of this application also provide a storage medium. The storage medium is a storage medium that stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned large model distributed inference acceleration method.

[0123] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] The distributed inference acceleration method and device for large models based on speculative sampling proposed in the embodiments of the present application input terminal prefix information into a draft small model for content generation to obtain multiple candidate tokens and the small model probability distribution corresponding to each candidate token, and use the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal-generated data and transmit it to the base station, so that the base station uses the target large model to generate the large model probability distribution of each candidate token according to the terminal-generated data, and classifies the candidate tokens into accepted tokens and / or rejected tokens. If the large model probability distribution corresponding to the first rejected token is received, resample the first rejected token to obtain a resampled token, write the resampled token into the parameter sequence, and update the terminal prefix information according to the resampled token, and iterate until the end condition is triggered to obtain a complete inference result. The embodiments described in the embodiments of the present application further decompose the verification process of candidate tokens, place the rejection / reception process of candidate tokens at the base station, and place the resampling process after rejection on the terminal device. Thus, the terminal device does not need to upload the small model probability distribution corresponding to each candidate token, and only needs to upload the corresponding small model probability value to participate in the rejection / reception judgment process. The base station only needs to return the large model probability distribution of the first rejected candidate token obtained by parallel decoding to the terminal device for resampling, and return the result of resampling to the base station for the next iteration process. In this way, the uplink transmission data volume from the terminal device to the base station is significantly reduced. Even if the downlink data volume of the base station increases slightly, the overall data volume is reduced, and the collaborative inference efficiency is still high.

[0125] The embodiments described in the embodiments of the present application are for more clearly explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0126] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown, or combine certain steps, or different steps.

[0127] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and their appropriate combinations.

[0129] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0130] It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or a similar expression means any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0131] In several embodiments provided by this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0132] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] In addition, in each embodiment of this application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store programs.

[0135] The preferred embodiments of the embodiments of this application have been described above with reference to the accompanying drawings, and thus do not limit the scope of rights of the embodiments of this application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall be within the scope of rights of the embodiments of this application.

Claims

1. A method for accelerating large model distributed inference based on speculative sampling, characterized in that, Applied to a terminal device on which a draft small model is deployed, the method includes: Inputting terminal prefix information into the draft small model for content generation to obtain a plurality of candidate tokens and the small model probability distribution corresponding to each candidate token; Transmitting the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal-generated data to a base station, so that the base station uses a target large model to generate the large model probability distribution of each candidate token according to the terminal-generated data, and classifies the candidate tokens into accepted tokens and / or rejected tokens, where the number of parameters of the target large model is higher than that of the draft small model; If the large model probability distribution corresponding to the first rejected token is received, resample the first rejected token to obtain a resampled token, write the resampled token into the parameter sequence, and update the terminal prefix information according to the resampled token; Iteratively execute until an end condition is triggered to obtain an inference result.

2. The large model distributed inference acceleration method based on speculative sampling according to claim 1, wherein The step of if the large model probability distribution corresponding to the first rejected token is received, resampling the first rejected token to obtain a resampled token includes: Obtaining the small model probability distribution corresponding to the first rejected token, and calculating the difference between the corresponding large model probability distribution and the small model probability distribution; Performing non-negative normalization on the difference to obtain a resampling probability distribution, and selecting the resampled token from the vocabulary based on the resampling probability distribution.

3. The method for accelerating distributed inference of large models based on speculative sampling according to claim 1, wherein Before the step of iteratively execute until an end condition is triggered to obtain an inference result, the method further includes: If the base station token generated by the base station according to the candidate token is received, initializing the parameter sequence, and updating the terminal prefix information according to the plurality of candidate tokens and the base station token.

4. A distributed inference acceleration method for large models based on speculative sampling, characterized in that, Applied to a base station on which a target large model is deployed, the method includes: Receiving terminal-generated data from a terminal device, and obtaining the current parameter sequence, each candidate token, and the small model probability value corresponding to the candidate token from the terminal-generated data, where the candidate token is generated by a draft small model in the terminal device according to terminal prefix information, and the number of parameters of the target large model is higher than that of the draft small model; Obtaining base station prefix information based on the parameter sequence, and inputting the base station prefix information into the target large model for content generation to obtain a base station token and the large model probability distribution corresponding to each candidate token; Classifying the candidate tokens into accepted tokens and / or rejected tokens according to the corresponding large model probability distribution and the small model probability value, and obtaining base station-generated data according to the large model probability distribution corresponding to the first rejected token or the base station token, and sending the base station-generated data to the terminal device; Iteratively execute until the end condition is triggered to obtain the inference result.

5. The method for accelerating distributed inference of large models based on speculative sampling according to claim 4, wherein, Obtaining updated prefix information based on the parameter sequence includes: Obtain the base station prefix information corresponding to the previous iteration and multiple of the received tokens, where the initial value of the base station prefix information is the same as the initial value of the terminal prefix information; Concatenate multiple of the received tokens and the parameter sequence after the base station prefix information in sequence to update the base station prefix information.

6. The method for accelerating distributed inference of large models based on speculative sampling according to claim 4, wherein Classifying the candidate tokens into received tokens according to the corresponding large model probability distribution and the small model probability value, including: Obtain the large model probability value corresponding to each candidate token from the large model probability distribution; For the candidate token, if the corresponding small model probability value is less than or equal to the large model probability value, use the candidate token as the received token.

7. The method for accelerating large model distributed inference based on speculative sampling according to claim 4, wherein Classifying the candidate tokens into rejected tokens according to the corresponding large model probability distribution and the small model probability value, including: Obtain the large model probability value corresponding to each candidate token from the large model probability distribution; For the candidate token, if the corresponding small model probability value is greater than the large model probability value, generate a random probability value within a preset range. If the random probability value is greater than or equal to the quotient of the large model probability value and the small model probability value, use the candidate token as the rejected token, otherwise use the candidate token as the received token.

8. A large model distributed inference acceleration device based on speculative sampling, characterized in that, Applied to a terminal device with a draft small model deployed thereon, the device includes: Candidate content generation module: used to input terminal prefix information into the draft small model for content generation to obtain multiple candidate tokens and the small model probability distribution corresponding to each candidate token; Parameter sending module: used to transmit the current parameter sequence, each candidate token, and the small model probability value of the candidate token in the corresponding small model probability distribution as terminal-generated data to the base station, so that the base station uses a target large model to generate the large model probability distribution of each candidate token according to the terminal-generated data, and classify the candidate tokens into received tokens and / or rejected tokens, where the number of parameters of the target large model is higher than that of the draft small model; Resampling module: used to resample the first rejected token if the large model probability distribution corresponding to the first rejected token is received to obtain a resampled token, write the resampled token into the parameter sequence, and update the terminal prefix information according to the resampled token; Terminal iteration module: used to iteratively execute until the end condition is triggered to obtain the inference result.

9. A large model distributed inference acceleration device based on speculative sampling, characterized in that, Applied to a base station with a target large model deployed thereon, the device includes: Data receiving module: It is used to receive the terminal-generated data from the terminal device, obtain the current parameter sequence, each candidate token, and the small model probability value corresponding to the candidate token from the terminal-generated data. The candidate token is generated by the draft small model in the terminal device according to the terminal prefix information, and the number of parameters of the target large model is higher than that of the draft small model; Large model parsing module: It is used to obtain the base station prefix information based on the parameter sequence, and input the base station prefix information into the target large model for content generation to obtain the base station token and the large model probability distribution corresponding to each candidate token; Acceptance / rejection module: It is used to classify the candidate tokens into acceptance tokens and / or rejection tokens according to the corresponding large model probability distribution and the small model probability value, and obtain the base station generated data according to the large model probability distribution corresponding to the first rejection token or the base station token, and send the base station generated data to the terminal device; Base station iteration module: It is used to perform iterative execution until the end condition is triggered to obtain the inference result.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the large model distributed inference acceleration method based on speculative sampling according to any one of claims 1 to 7.

11. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the large model distributed inference acceleration method based on speculative sampling according to any one of claims 1 to 7.

Images