Big language model-based reasoning method and device, electronic equipment and storage medium

Abstract

The invention provides an inference method and device based on a large language model, electronic equipment and a storage medium. The method comprises the following steps: sending a received reasoning request to a cloud, so that the cloud generates a key value cache corresponding to the reasoning request based on a first model, and quantifying the key value cache to obtain a first quantized key value cache; receiving a first quantized key value cache returned by the cloud, and generating a candidate token sequence corresponding to the reasoning request based on the first quantized key value cache and a second model; and sending the candidate token sequence to the cloud, so that the cloud verifies the candidate token sequence by using the key value cache and the first model to obtain a reasoning result. According to the embodiment of the invention, the model with a large parameter quantity is deployed at the cloud, the model with a small parameter quantity is deployed at the mobile terminal, and after the mobile terminal receives the reasoning request, the reasoning result is finally given through cooperative processing with the cloud, so that the reasoning accuracy is improved.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to an inference method, device, electronic device and storage medium based on a large language model. Background Art

[0002] The enhancement of the capabilities and performance of large models is inseparable from the increase in the scale of large model parameters. The increase in the scale of large model parameters puts forward new requirements for the hardware for deploying large models. The deployment of large models requires hardware accelerators with stronger computing power, larger memory and bandwidth.

[0003] As large models are increasingly used on the consumer side, AI desktops and mobile phones will become more popular. However, mobile hardware accelerators generally have weak computing resources, small storage and bandwidth. Using large models with small parameters or quantized models often leads to a loss of model performance, which in turn leads to inaccurate inference results. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an inference method, device, electronic device and storage medium based on a large language model, so as to achieve a large language model with a large number of parameters when performing inference through a mobile terminal, thereby improving the accuracy of inference.

[0005] In a first aspect, an embodiment of the present application provides an inference method based on a large language model, which is applied to a mobile terminal, the mobile terminal communicates with the cloud, a first model is deployed in the cloud, and a second model is deployed in the mobile terminal; wherein the parameter amount of the first model is greater than the parameter amount of the second model; the inference method includes:

[0006] Sending the received inference request to the cloud, so that the cloud generates a key-value cache corresponding to the inference request based on the first model, and quantizes the key-value cache to obtain a first quantized key-value cache;

[0007] Receive the first quantized key-value cache returned by the cloud, and generate a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model;

[0008] The candidate token sequence is sent to the cloud, so that the cloud verifies the candidate token sequence using the key-value cache and the first model to obtain an inference result.

[0009] The embodiment of the present application deploys a model with a large number of parameters in the cloud and a model with a small number of parameters in the mobile terminal. After the mobile terminal receives the inference request, it collaborates with the cloud to finally give the inference result, thereby improving the accuracy of the inference.

[0010] In any embodiment, generating a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model includes:

[0011] quantizing the key-value cache after the first quantization again to obtain the key-value cache after the second quantization;

[0012] Generate an initial token sequence corresponding to the inference request based on the second quantized key-value cache and the second model;

[0013] The initial token sequence is verified using the first quantized key-value cache and the second model to obtain a candidate token sequence.

[0014] In an embodiment of the present application, the mobile terminal re-quantizes the key-value cache after the first quantization to obtain a second quantized key-value cache with a smaller parameter amount. In the reasoning stage, the mobile terminal obtains a candidate token sequence through two reasonings, which can improve the reasoning efficiency.

[0015] In any embodiment, the initial token sequence includes multiple initial tokens and a first probability corresponding to each initial token; and verifying the initial token sequence using the first quantized key-value cache and the second model includes:

[0016] Calculate the second probability corresponding to each initial token in parallel using the first quantized key-value cache and the second model;

[0017] If the second probability is greater than the first probability, the corresponding initial token is used as the result of the second model; otherwise, the corresponding initial token is discarded and resampled until a complete candidate token sequence is generated.

[0018] In an embodiment of the present application, during the reasoning process, the mobile terminal uses the first quantized key-value cache and the second model to calculate the second probability corresponding to the initial token, and performs verification based on the second probability and the first probability corresponding to the initial token to obtain a candidate token sequence, thereby improving the reasoning rate.

[0019] In a second aspect, an embodiment of the present application provides another reasoning method based on a large language model, which is applied to the cloud, the cloud communicates with the mobile terminal, the cloud is deployed with a first model, and the mobile terminal is deployed with a second model; wherein the parameter amount of the first model is greater than the parameter amount of the second model; the reasoning method includes:

[0020] Receiving an inference request sent by a mobile terminal;

[0021] Generate a key-value cache corresponding to the inference request based on the first model;

[0022] quantizing the key-value cache to obtain a first quantized key-value cache, and sending the first quantized key-value cache to the mobile terminal, so that the mobile terminal generates a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model;

[0023] A candidate token sequence sent by the mobile terminal is received, and the candidate token sequence is verified using the key-value cache and the first model to obtain an inference result.

[0024] The embodiment of the present application deploys a model with a large number of parameters in the cloud and a model with a small number of parameters in the mobile terminal. After the mobile terminal receives the inference request, it collaborates with the cloud to finally give the inference result, thereby improving the accuracy of the inference.

[0025] In any embodiment, the candidate token sequence includes multiple candidate tokens and a second probability corresponding to each candidate token; the candidate token sequence is verified using the key-value cache and the first model, including:

[0026] The key-value cache and the first model are used to calculate multiple candidate tokens in parallel to obtain the third probability corresponding to each candidate token;

[0027] If the third probability is greater than the second probability, the corresponding candidate token is used as the result of the first model; otherwise, the corresponding candidate token is discarded and resampled until a complete inference result is generated.

[0028] In the embodiment of the present application, multiple candidate tokens are calculated in parallel in the first model through key-value caching to obtain the third probability corresponding to each candidate token; and the third probability and the second probability corresponding to the candidate token are used for verification, so that the mobile terminal can enjoy the reasoning effect of the large model on the cloud.

[0029] In a third aspect, an embodiment of the present application provides another reasoning method based on a large language model, which is collaboratively reasoned by a cloud and a mobile terminal; a first model is deployed on the cloud, and a second model is deployed on the mobile terminal; wherein the parameter amount of the first model is greater than the parameter amount of the second model; the method comprises:

[0030] The mobile terminal sends the received inference request to the cloud;

[0031] The cloud generates a key-value cache corresponding to the inference request based on the first model;

[0032] The cloud quantizes the key-value cache to obtain a first quantized key-value cache, and sends the first quantized key-value cache to the mobile terminal;

[0033] The mobile terminal generates a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model, and sends the candidate token sequence to the cloud;

[0034] The cloud uses the key-value cache and the first model to verify the candidate token sequence to obtain the inference result.

[0035] The embodiment of the present application deploys a model with a large number of parameters in the cloud and a model with a small number of parameters in the mobile terminal. After the mobile terminal receives the inference request, it collaborates with the cloud to finally give the inference result, thereby improving the accuracy of the inference.

[0036] In any embodiment, the first model and the second model are compatible.

[0037] The embodiment of the present application sets the first model and the second model to be compatible so that data communication can be performed between the two models to achieve the purpose of collaborative reasoning between the two models.

[0038] In any embodiment, the cloud and the mobile terminal encrypt the data to be transmitted before transmitting the data.

[0039] The embodiments of the present application improve the security of data transmission by encrypting the data to be transmitted.

[0040] In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory, and a bus, wherein:

[0041] The processor and the memory communicate with each other via the bus;

[0042] The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect or the second aspect by calling the program instructions.

[0043] In a fifth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, including:

[0044] The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the method of the first aspect or the second aspect.

[0045] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which, when read and executed by a processor, execute the method of the first aspect or the second aspect.

[0046] In the seventh aspect, an embodiment of the present application provides an inference system based on a large language model, comprising a cloud and a mobile terminal; wherein the cloud is deployed with a first model, and the mobile terminal is deployed with a second model; wherein the number of parameters of the first model is greater than the number of parameters of the second model; the mobile terminal is capable of executing the method described in the first aspect, and the cloud is capable of executing the method described in the second aspect.

[0047] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a reasoning method based on a large language model provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a reasoning method provided in an embodiment of the present application;

[0051] Figure 3 A flowchart of another reasoning method based on a large language model provided in an embodiment of the present application;

[0052] Figure 4 A signaling interaction diagram of another reasoning method based on a large language model provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of an inference device based on a large language model provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the structure of another reasoning device based on a large language model provided in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application;

[0056] Figure 8 A structural diagram of an inference system based on a large language model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0059] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0060] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0061] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0062] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0063] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0064] For ease of understanding, the terms involved in the embodiments of the present application are now explained:

[0065] Large Language Model (LLM): Large language models usually refer to models trained using deep learning techniques, such as the Transformer architecture. These models are based on large-scale text datasets, such as the Generative Pre-trained Transformer (GPT) series (GPT-3, GPT-4, etc.), Bidirectional Encoder Representations from Transformers (BERT), etc. Their main feature is that they have powerful language understanding and generation capabilities, and can be used for a variety of natural language processing tasks such as text generation, text classification, question answering, and summarization. Large language models can be used to generate new text content as well as to understand and interpret input natural language text.

[0066] Key-value cache: It is a cache mechanism for storing key-value pairs. By caching the calculated or accessed data, it reduces the repeated calculation or access to the original data, thereby improving the data access speed and system performance. In large language models, key-value cache is mainly used to store the key (Key) and value (Value) vectors in the attention mechanism to avoid recalculating these vectors every time a new token is generated, thereby significantly reducing the amount of calculation and improving the reasoning speed.

[0067] Quantize the key-value cache: The process of quantizing the KV cache is mainly to reduce its occupancy in the memory or video memory, thereby improving the reasoning efficiency of large language models.

[0068] Speculative sampling: It is a method to accelerate the reasoning of large models by parallel computing and reducing the number of serial calls without losing the generation effect. The core is to use the small model to perform preliminary autoregressive sampling to generate candidate token sequences, and then input these candidate sequences into the large model (Target model) for verification and correction. Since the small model has a small amount of calculation and can handle most simple token generation tasks, it can significantly reduce the number of calls to the large model and improve the reasoning speed.

[0069] In an embodiment of the present application, the above-mentioned process of generating answers to questions can be performed through a generative large language model, and the part of obtaining the generative large language model and adjusting the model parameters involves artificial intelligence (AI), machine learning (ML) technology, and natural language processing, and is designed based on feature extraction, machine learning, and natural language processing in artificial intelligence.

[0070] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0071] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model (Pre-Trained Model), also known as the large model or basic model, can be widely used in downstream tasks in various major directions of artificial intelligence after fine-tuning. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0072] Specifically, the pre-trained model refers to a deep neural network (DNN) with large parameters, which is trained on massive unlabeled data. The function approximation ability of the large-parameter DNN is used to enable the PTM to extract common features from the data. After fine tuning, parameter efficient fine tuning (PEFT), prompt-tuning and other technologies, it is suitable for downstream tasks. Therefore, the pre-trained model can achieve ideal results in few-shot or zero-shot scenarios. PTM can be divided into language models (ELMO, BER T, GPT), visual models (swintransformer, ViT, V-MOE), speech models (VALL-E), multimodal models (ViBERT, CLIP, Flamingo, Gato), etc. according to the data modality processed. Among them, the multimodal model refers to a model that establishes two or more data modality feature representations. The pre-trained model is an important tool for outputting artificial intelligence generated content (AIGC), and can also be used as a general interface to connect multiple specific task models.

[0073] Machine learning is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0074] Natural language processing is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between people and computers using natural language. Natural language processing involves natural language, which is the language people use in daily life, and is closely related to linguistic research; it also involves computer science and mathematics, and is an important technology for model training in the field of artificial intelligence. The pre-trained model is developed from the Large Language Model (LLM) in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0075] The solution provided in the embodiments of the present application mainly relates to the application of artificial intelligence in question answering. The generative large language model (Generative Large Language Model) in the embodiments of the present application is a deep learning model that can be used for natural language processing tasks such as text generation, summarization, translation, etc.

[0076] The computing power, storage and bandwidth of the computing resources of mobile terminals are insufficient to support the reasoning deployment of large models, and the use of models with smaller parameters or large models after quantization often results in model performance degradation. In order to experience the higher performance advantages of large models with large parameters on mobile terminals, the embodiments of the present application deploy small-parameter models on mobile terminals and large-parameter models on the cloud, and the mobile terminal and the cloud collaborate in reasoning, combining the advantages of different hardware resources of the cloud and mobile terminals, to accelerate large model reasoning and improve the large model reasoning service experience of mobile terminals.

[0077] Figure 1 A flowchart of a reasoning method based on a large language model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method is applied to a mobile terminal, which can be a desktop computer, a laptop computer, a tablet computer, a smart phone, and a smart wearable device. The mobile terminal can communicate with the cloud. A first model is deployed in the cloud, and a second model is deployed in the mobile terminal. The parameter amount of the first model is greater than the parameter amount of the second model. For example: a model with a parameter amount greater than 70B can be deployed as a first model in the cloud, and a model not greater than 70B can be deployed as a second model in the mobile terminal. For example, the first model is usually a large model of 70B, which is not easy for a mobile terminal to fully store weights and kv cache, or a large model with an ultra-long context (such as more than 128K) reasoning capability. The second model is a quantized small model, which usually refers to a model of less than 7B that is convenient for deployment in a mobile terminal. The quantization method includes AWQ, LLM.int8, SmoothQuant, etc.

[0078] The method includes:

[0079] Step 101: sending the received inference request to the cloud, so that the cloud generates a key-value cache corresponding to the inference request based on a first model, and quantizes the key-value cache to obtain a first quantized key-value cache;

[0080] Step 102: Receive the first quantized key-value cache returned by the cloud, and generate a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model;

[0081] Step 103: Send the candidate token sequence to the cloud, so that the cloud verifies the candidate token sequence using the key-value cache and the first model to obtain an inference result.

[0082] In the specific implementation process, the reasoning request is sent by the user to the mobile terminal, which can be input through the input device on the mobile terminal, for example, it can be input text or voice. The reasoning request can include a stem and a question, where the stem is a broad topic or background information that provides context for the question. The stem may contain information such as text, pictures, and charts, which help understand the background and requirements of the question. The question is a question about a specific aspect or detail mentioned in the stem. Questions usually require specific answers or solutions.

[0083] For example: Subject: In recent years, more and more people have begun to pay attention to healthy eating to prevent diseases and maintain good health. Question: What are the basic principles of healthy eating?

[0084] It should be noted that to answer these questions, it is necessary to analyze and reason based on the information in the question stem. In the embodiment of the present application, the answers to the questions are mainly generated based on the collaborative reasoning of the second model of the mobile terminal and the first model in the cloud.

[0085] The following content is the collaborative reasoning process between the mobile terminal and the cloud:

[0086] After receiving the inference request, the mobile terminal can tokenize the inference request and then send it to the cloud. After receiving the inference request, the cloud uses the first model to generate a key-value cache (kv cache) corresponding to the inference request. During the processing, the first model calculates the key (K) and value (V) vectors of each token and caches these vectors. Therefore, the full amount of kv cache is stored in the cloud. Generally, large models with a large number of parameters, such as 70B, etc., which fully store weights in fp16 data format require more than 100GB of memory. For ultra-long context situations, the storage occupied by the KV cache is 5 times that of the model weight storage. Therefore, the deployment of large models has greater hardware requirements, and large models can be deployed in the cloud.

[0087] After obtaining the kv cache, the cloud quantizes the kv cache to obtain a first quantized key-value cache. Then the first quantized key-value cache is sent to the mobile terminal. Among them, quantization is a means of reducing the kv cache, including quantization of the kvcache data format and other kv cache compression and sparseness technologies, including but not limited to: vertex quantization, symmetric quantization, asymmetric quantization, layer or channel quantization and other methods.

[0088] After receiving the first quantized key-value cache, the mobile terminal generates a candidate token sequence corresponding to the inference request according to the first quantized key-value cache and the second model.

[0089] The mobile terminal sends the candidate token sequence to the cloud, and the cloud verifies the candidate token sequence (including k tokens) in parallel to obtain the inference result.

[0090] After obtaining the inference results, the cloud returns the inference results to the mobile terminal, and the mobile terminal displays the inference results to the user.

[0091] It should be emphasized that in the specific implementation methods of this application, the relevant data of questions and answers involved, such as reasoning requests and reasoning results, etc., when the embodiments of this application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0092] The embodiment of the present application deploys a model with a large number of parameters in the cloud and a model with a small number of parameters in the mobile terminal. After the mobile terminal receives the inference request, it collaborates with the cloud to finally give the inference result, thereby improving the accuracy of the inference.

[0093] On the basis of the above embodiment, generating a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model includes:

[0094] quantizing the key-value cache after the first quantization again to obtain the key-value cache after the second quantization;

[0095] Generate an initial token sequence corresponding to the inference request based on the second quantized key-value cache and the second model;

[0096] The initial token sequence is verified using the first quantized key-value cache and the second model to obtain a candidate token sequence.

[0097] In a specific implementation process, after receiving the first quantized key-value cache, the mobile terminal may further quantize the first quantized key-value cache to obtain a second quantized key-value cache. It is understandable that an eviction strategy may be selected during the re-quantization, and the eviction strategy includes a streaming cache, H20, and the like.

[0098] Therefore, there are two levels of models in the mobile terminal, namely: a primary model consisting of the key-value cache after the second quantization and the second model, and a secondary model consisting of the key-value cache after the first quantization and the second model.

[0099] In the inference stage, the mobile terminal uses the inference request as the input of the first-level model, and the first-level model generates k tokens through inference. Specifically, after generating the first token, the key feature corresponding to the token is spliced ​​to the key feature set in the second quantized key value cache, and the second token is predicted again based on this spliced ​​key feature set; after predicting the second token, the key feature corresponding to the second token is spliced ​​to the existing key feature set (i.e., the key feature in the second quantized key value cache and the splicing result with the key feature of the first token), and the third token is predicted again based on this spliced ​​key feature set; after obtaining the third token, the key feature corresponding to the token is spliced ​​to the current existing key feature set (i.e., the key feature set in the second quantized key value cache and the splicing result with the key features of the first token and the second token), and the next token is predicted again based on this spliced ​​key feature set; and so on, until k tokens are output. Among them, the value of k can represent a complete answer or a length upper limit.

[0100] Similarly, after predicting the first token of the answer, the value feature corresponding to the token is spliced ​​to the value feature set in the second quantized key-value cache, and the second token is predicted again based on the spliced ​​value feature set; after predicting the second token, the value feature corresponding to the token is spliced ​​to the current existing value feature set (i.e., the value feature in the second quantized key-value cache and the splicing result with the value feature of the first token), and the third token is predicted again based on the spliced ​​value feature set; after obtaining the third token, the value feature corresponding to the token is spliced ​​to the current existing value feature set (i.e., the value feature set in the second quantized key-value cache and the splicing result with the value features of the first token and the second token), and the next token is predicted again based on the spliced ​​value feature set; and so on, until k tokens are output. The value of k can represent the complete answer or the upper limit of the length.

[0101] After obtaining k tokens, the k tokens are used as the input of the secondary model, and the secondary model verifies the input in parallel. The so-called parallel verification means that the k tokens are input into the secondary model in parallel, and the secondary model verifies each token in parallel to obtain a candidate token sequence. After obtaining the candidate token sequence, the mobile terminal sends the candidate token sequence to the cloud, where the candidate token sequence includes k tokens. The cloud verifies the k candidate tokens in parallel to obtain the final inference result.

[0102] In an embodiment of the present application, the mobile terminal re-quantizes the key-value cache after the first quantization to obtain a second quantized key-value cache with a smaller parameter amount. In the reasoning stage, the mobile terminal obtains a candidate token sequence through two reasonings, which can improve the reasoning efficiency.

[0103] Based on the above embodiment, the initial token sequence includes multiple initial tokens and a first probability corresponding to each initial token. The first probability is calculated when the initial token sequence is generated. The specific method of verifying the initial token sequence using the first quantized key-value cache and the second model includes:

[0104] Calculate the second probability corresponding to each initial token in parallel using the first quantized key-value cache and the second model;

[0105] If the second probability is greater than the first probability, the corresponding initial token is used as the result of the second model; otherwise, the corresponding initial token is discarded and resampled until a complete candidate token sequence is generated.

[0106] In the specific implementation process, Figure 2 A schematic diagram of a reasoning method provided in an embodiment of the present application, such as Figure 2 As shown. The embodiment of the present application adopts a three-level speculative sampling decoding reasoning method, including first-level reasoning, second-level reasoning and third-level reasoning. Among them, the first-level reasoning and the second-level reasoning are performed in the mobile terminal, and the third-level reasoning is performed in the cloud. The first-level reasoning is the reasoning of the key-value cache after the second quantization and the second model, the second-level reasoning is the reasoning of the key-value cache after the first quantization and the second model, and the third-level reasoning is the reasoning of the full key-value cache and the first model.

[0107] The mobile terminal predicts the initial token sequence in parallel based on the second model and the first quantized key-value cache to obtain a second probability of predicting the same K initial tokens. The prediction results of the primary model and the secondary model are compared. If the probability predicted by the secondary model of the token is greater than the probability predicted by the primary model, the initial token is accepted and used as a candidate token. Otherwise, it is rejected with a certain probability and resampled until k candidate tokens are obtained.

[0108] Similarly, the cloud's verification process for candidate token sequences is similar to the above verification process. The cloud performs parallel predictions on the candidate token sequence based on the first model and the key-value cache to obtain the third probability of predicting the same k candidate tokens. Compare the results predicted by the secondary model and the first model. If the probability predicted by the first model of the token is greater than the probability predicted by the secondary model, the candidate token is accepted and the initial candidate is used as a token in the inference result. Otherwise, it is rejected with a certain probability and resampled until a complete inference result is obtained.

[0109] In an embodiment of the present application, during the reasoning process, the mobile terminal uses the first quantized key-value cache and the second model to calculate the second probability corresponding to the initial token, and performs verification based on the second probability and the first probability corresponding to the initial token to obtain a candidate token sequence, thereby improving the reasoning rate.

[0110] On the basis of the above embodiments, in order to ensure the security of data transmission between the cloud and the mobile terminal, a digital signature method is used for data security protection. The public key is saved in the cloud and the mobile terminal, and the data is hashed to generate a hash value before being transmitted to the cloud or the mobile terminal, and the hash value is signed with a private key to generate a digital signature. The data recipient uses the public key to verify the data signature.

[0111] On the basis of the above embodiments, the first model deployed on the cloud and the second model deployed on the mobile terminal need to have a certain degree of compatibility, that is, using the same series of models, for example, using LLaMA-70B on the cloud and LLaMA-7B on the mobile terminal.

[0112] Figure 3 A flowchart of another reasoning method based on a large language model provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method is applied to the cloud, which is connected to the mobile terminal by wired communication or wireless communication. A first model is deployed in the cloud, and a second model is deployed in the mobile terminal. The parameter amount of the first model is greater than the parameter amount of the second model, for example: the parameter amount of the first model is a model greater than 70B, and the parameter amount of the second model is a model of 7B. The method includes:

[0113] Step 301: receiving an inference request sent by a mobile terminal;

[0114] Step 302: Generate a key-value cache corresponding to the inference request based on the first model;

[0115] Step 303: quantize the key-value cache to obtain a first quantized key-value cache, and send the first quantized key-value cache to the mobile terminal, so that the mobile terminal generates a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model;

[0116] Step 304: Receive the candidate token sequence sent by the mobile terminal, and verify the candidate token sequence using the key-value cache and the first model to obtain an inference result.

[0117] In the specific implementation process, the reasoning request is sent by the user to the mobile terminal, which can be input through the input device on the mobile terminal, for example, it can be input text or voice. The reasoning request can include a stem and a question, where the stem is a broad topic or background information that provides context for the question. The stem may contain information such as text, pictures, and charts, which help understand the background and requirements of the question. The question is a question about a specific aspect or detail mentioned in the stem. Questions usually require specific answers or solutions.

[0118] For example: Subject: In recent years, more and more people have begun to pay attention to healthy eating to prevent diseases and maintain good health. Question: What are the basic principles of healthy eating?

[0119] It should be noted that to answer these questions, it is necessary to analyze and reason based on the information in the question stem. In the embodiment of the present application, the answers to the questions are mainly generated based on the collaborative reasoning of the second model of the mobile terminal and the first model in the cloud.

[0120] The following content is the collaborative reasoning process between the mobile terminal and the cloud:

[0121] After receiving the inference request, the mobile terminal can tokenize the inference request and then send it to the cloud. After receiving the inference request, the cloud uses the first model to generate a key-value cache (kv cache) corresponding to the inference request. During the processing, the first model calculates the key (K) and value (V) vectors of each token and caches these vectors. Therefore, the full amount of kv cache is stored in the cloud. Generally, large models with a large number of parameters, such as 70B, etc., which fully store weights in fp16 data format require more than 100GB of memory. For ultra-long context situations, the storage occupied by the KV cache is 5 times that of the model weight storage. Therefore, the deployment of large models has greater hardware requirements, and large models can be deployed in the cloud.

[0122] After obtaining the kv cache, the cloud quantizes the kv cache to obtain a first quantized key-value cache. Then the first quantized key-value cache is sent to the mobile terminal. There are many methods for quantizing the kv cache, including but not limited to: vertex quantization, symmetric quantization, asymmetric quantization, layer or channel quantization, etc.

[0123] After receiving the first quantized key-value cache, the mobile terminal generates a candidate token sequence corresponding to the inference request according to the first quantized key-value cache and the second model.

[0124] The mobile terminal sends the candidate token sequence to the cloud, and the cloud verifies the candidate token sequence (including k tokens) in parallel to obtain the inference result. It should be noted that the specific steps of verifying the candidate token sequence in the cloud can be referred to the above embodiment, which will not be repeated here.

[0125] After obtaining the inference results, the cloud returns the inference results to the mobile terminal, and the mobile terminal displays the inference results to the user.

[0126] The embodiment of the present application deploys a model with a large number of parameters in the cloud and a model with a small number of parameters in the mobile terminal. After the mobile terminal receives the inference request, it collaborates with the cloud to finally give the inference result, thereby improving the accuracy of the inference.

[0127] Figure 4 Another signaling interaction diagram of a reasoning method based on a large language model provided in an embodiment of the present application is as follows: Figure 4 As shown. Through collaborative reasoning between the cloud and mobile terminals, the first model is deployed in the cloud and the second model is deployed in the mobile terminal. Model reasoning is divided into two stages, namely the prefill stage and the decoding stage. The Prefill stage, also known as the preprocessing or pre-filling stage, is the beginning stage of the model reasoning process. In this stage, the model mainly processes input data and prepares computing resources to provide the necessary input and computing environment for the subsequent Decoding stage. The Decoding stage, also known as the decoding or generation stage, is the core stage of the model reasoning process. In this stage, the model gradually generates output sequences based on the input and computing environment provided by the Prefill stage.

[0128] The following is an introduction to the specific implementation steps of the Prefill stage and the Decoding stage:

[0129] Step 401: The mobile terminal receives an inference request; it should be noted that the relevant explanation of the inference request can be found in the above embodiment and will not be repeated here.

[0130] Step 402: The mobile terminal sends an inference request to the cloud;

[0131] Step 403: The cloud generates a key-value cache corresponding to the inference request based on the first model;

[0132] Step 404: storing the key-value cache in the cloud, and quantizing the key-value cache to obtain a first quantized first key-value cache; the quantization method can refer to the above embodiment;

[0133] Step 405: the cloud sends the first quantized first key value cache to the mobile terminal;

[0134] Step 406: The mobile terminal quantizes the first quantized key-value cache again to obtain a second quantized key-value cache;

[0135] Step 407: the mobile terminal uses the second quantized key-value cache and the second model to infer the inference request and generate an initial token sequence; the method for generating the initial token sequence is as described in the above embodiment;

[0136] Step 408: The mobile terminal verifies the initial token sequence using the first quantized key value cache and the second model to obtain a candidate token sequence;

[0137] Step 409: The mobile terminal sends the candidate token sequence to the cloud;

[0138] Step 410: The cloud uses the key-value cache and the first model to verify the candidate token sequence and obtain an inference result.

[0139] Step 411: The cloud returns the inference result to the mobile terminal, and the mobile terminal displays the inference result to the user.

[0140] The embodiment of the present application adopts a multi-level speculative sampling coding method, and uses the sufficient computing and storage resources of the cloud to realize the collaborative reasoning of the cloud and mobile terminal large models. Speculative sampling coding uses the large model to parallelly verify the small model to generate tokens, reducing the delay of large model reasoning. The embodiment of the present application uses the multi-level features of speculative sampling coding, combined with the advantages of different hardware resources of the cloud and mobile terminals, to accelerate the reasoning of large models and improve the mobile terminal large model reasoning service experience.

[0141] Figure 5 The schematic diagram of the structure of a reasoning device based on a large language model provided in the embodiment of the present application, the device can be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above Figure 1 The method embodiment corresponds to and can be executed Figure 1The various steps involved in the method embodiment and the specific functions of the device can be found in the above description. To avoid repetition, detailed description is appropriately omitted here. The device includes: a first sending module 501, a first receiving module 502 and a second sending module 503, wherein:

[0142] The first sending module 501 is used to send the received inference request to the cloud, so that the cloud generates a key-value cache corresponding to the inference request based on the first model, and quantizes the key-value cache to obtain a first quantized key-value cache;

[0143] The first receiving module 502 is used to receive the first quantized key-value cache returned by the cloud, and generate a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model;

[0144] The second sending module 503 is used to send the candidate token sequence to the cloud, so that the cloud uses the key-value cache and the first model to verify the candidate token sequence to obtain an inference result.

[0145] Based on the above embodiment, the first receiving module 502 is specifically used for:

[0146] quantizing the first quantized key-value cache again to obtain a second quantized key-value cache;

[0147] generating an initial token sequence corresponding to the inference request based on the second quantized key-value cache and the second model;

[0148] The initial token sequence is verified using the first quantized key-value cache and the second model to obtain the candidate token sequence.

[0149] Based on the above embodiment, the initial token sequence includes multiple initial tokens and a first probability corresponding to each initial token; the first receiving module 502 is specifically used for:

[0150] Calculate the second probability corresponding to each of the initial tokens in parallel using the first quantized key-value cache and the second model;

[0151] If the second probability is greater than the first probability, the corresponding initial token is used as the result of the second model; otherwise, the corresponding initial token is discarded and resampled until a complete candidate token sequence is generated.

[0152] Figure 6This is a schematic diagram of another inference device based on a large language model provided in an embodiment of the present application. The device may be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above-mentioned Figure 3 The method embodiment corresponds to and can be executed Figure 3 The various steps involved in the method embodiment and the specific functions of the device can be found in the above description. To avoid repetition, detailed description is appropriately omitted here. The device includes: a second receiving module 601, a cache generation module 602, a quantification module 603 and a verification module 604, wherein:

[0153] The second receiving module 601 is used to receive the inference request sent by the mobile terminal;

[0154] The cache generation module 602 is used to generate a key-value cache corresponding to the reasoning request based on the first model;

[0155] The quantization module 603 is used to quantize the key-value cache to obtain a first quantized key-value cache, and send the first quantized key-value cache to the mobile terminal, so that the mobile terminal generates a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model;

[0156] The verification module 604 is used to receive the candidate token sequence sent by the mobile terminal, and verify the candidate token sequence using the key-value cache and the first model to obtain an inference result.

[0157] On the basis of the above embodiment, the candidate token sequence includes multiple candidate tokens and a second probability corresponding to each candidate token; the verification module 604 is specifically used to:

[0158] Utilizing the key-value cache and the first model to perform parallel calculations on the multiple candidate tokens, and obtaining a third probability corresponding to each of the candidate tokens;

[0159] If the third probability is greater than the second probability, the corresponding candidate token is used as the result of the first model; otherwise, the corresponding candidate token is discarded and resampled until a complete inference result is generated.

[0160] Figure 7 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device includes: a processor (processor) 701, a memory (memory) 702 and a bus 703; wherein,

[0161] The processor 701 and the memory 702 communicate with each other via the bus 703;

[0162] The processor 701 is used to call the program instructions in the memory 702 to execute the methods provided by the above-mentioned method embodiments, for example, including: sending the received inference request to the cloud, so that the cloud generates a key value cache corresponding to the inference request based on the first model, and quantizes the key value cache to obtain a first quantized key value cache; receiving the first quantized key value cache returned by the cloud, and generating a candidate token sequence corresponding to the inference request based on the first quantized key value cache and the second model; sending the candidate token sequence to the cloud, so that the cloud uses the key value cache and the first model to verify the candidate token sequence to obtain an inference result. Or,

[0163] Receive an inference request sent by the mobile terminal; generate a key value cache corresponding to the inference request based on the first model; quantize the key value cache to obtain a first quantized key value cache, and send the first quantized key value cache to the mobile terminal, so that the mobile terminal generates a candidate token sequence corresponding to the inference request based on the first quantized key value cache and the second model; receive the candidate token sequence sent by the mobile terminal, and verify the candidate token sequence using the key value cache and the first model to obtain an inference result.

[0164] The processor 701 may be an integrated circuit chip with signal processing capabilities. The processor 701 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It may implement or execute various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0165] The memory 702 may include but is not limited to random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), etc.

[0166] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: sending the received inference request to the cloud, so that the cloud generates a key-value cache corresponding to the inference request based on the first model, and quantizes the key-value cache to obtain a first quantized key-value cache; receiving the first quantized key-value cache returned by the cloud, and generating a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model; sending the candidate token sequence to the cloud, so that the cloud uses the key-value cache and the first model to verify the candidate token sequence to obtain an inference result. Or,

[0167] Receive an inference request sent by the mobile terminal; generate a key value cache corresponding to the inference request based on the first model; quantize the key value cache to obtain a first quantized key value cache, and send the first quantized key value cache to the mobile terminal, so that the mobile terminal generates a candidate token sequence corresponding to the inference request based on the first quantized key value cache and the second model; receive the candidate token sequence sent by the mobile terminal, and verify the candidate token sequence using the key value cache and the first model to obtain an inference result.

[0168] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: sending the received inference request to the cloud, so that the cloud generates a key-value cache corresponding to the inference request based on the first model, and quantizes the key-value cache to obtain a first quantized key-value cache; receiving the first quantized key-value cache returned by the cloud, and generating a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model; sending the candidate token sequence to the cloud, so that the cloud uses the key-value cache and the first model to verify the candidate token sequence to obtain an inference result. Or,

[0169] Receive an inference request sent by the mobile terminal; generate a key value cache corresponding to the inference request based on the first model; quantize the key value cache to obtain a first quantized key value cache, and send the first quantized key value cache to the mobile terminal, so that the mobile terminal generates a candidate token sequence corresponding to the inference request based on the first quantized key value cache and the second model; receive the candidate token sequence sent by the mobile terminal, and verify the candidate token sequence using the key value cache and the first model to obtain an inference result.

[0170] Figure 8 A structural diagram of a reasoning system based on a large language model provided in an embodiment of the present application, such as Figure 8 As shown, the system includes a cloud 801 and a mobile terminal 802, wherein the cloud 801 is deployed with a first model, and the mobile terminal 802 is deployed with a second model. The parameter quantity of the first model is greater than the parameter quantity of the second model. The cloud 801 and the mobile terminal 802 can collaboratively reason the methods provided in the above-mentioned embodiments, which will not be repeated here.

[0171] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0172] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0174] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0175] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A reasoning method based on a large language model, characterized in that: Applied to a mobile terminal, the mobile terminal communicates with a cloud, the cloud is deployed with a first model, and the mobile terminal is deployed with a second model; wherein the parameter amount of the first model is greater than the parameter amount of the second model; the method comprises: Sending the received inference request to the cloud, so that the cloud generates a key-value cache corresponding to the inference request based on the first model, and quantizing the key-value cache to obtain a first quantized key-value cache; Receiving the first quantized key-value cache returned by the cloud, and generating a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model; The candidate token sequence is sent to the cloud, so that the cloud verifies the candidate token sequence using the key-value cache and the first model to obtain an inference result.

2. The method according to claim 1, characterized in that The generating a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model includes: quantizing the first quantized key-value cache again to obtain a second quantized key-value cache; generating an initial token sequence corresponding to the inference request based on the second quantized key-value cache and the second model; The initial token sequence is verified using the first quantized key-value cache and the second model to obtain the candidate token sequence.

3. The method according to claim 2, characterized in that The initial token sequence includes multiple initial tokens and a first probability corresponding to each initial token; and the verifying the initial token sequence by using the first quantized key-value cache and the second model includes: Calculate the second probability corresponding to each of the initial tokens in parallel using the first quantized key-value cache and the second model; If the second probability is greater than the first probability, the corresponding initial token is used as the result of the second model; otherwise, the corresponding initial token is discarded and resampled until a complete candidate token sequence is generated.

4. A reasoning method based on a large language model, characterized in that: Applied to a cloud, the cloud communicates with a mobile terminal, the cloud is deployed with a first model, and the mobile terminal is deployed with a second model; wherein the parameter amount of the first model is greater than the parameter amount of the second model; the method comprises: Receiving an inference request sent by the mobile terminal; Generate a key-value cache corresponding to the inference request based on the first model; quantizing the key-value cache to obtain a first quantized key-value cache, and sending the first quantized key-value cache to the mobile terminal, so that the mobile terminal generates a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model; The candidate token sequence sent by the mobile terminal is received, and the candidate token sequence is verified using the key-value cache and the first model to obtain an inference result.

5. The method according to claim 4, characterized in that The candidate token sequence includes multiple candidate tokens and a second probability corresponding to each candidate token; and the verifying the candidate token sequence by using the key-value cache and the first model includes: Utilizing the key-value cache and the first model to perform parallel calculations on the multiple candidate tokens, and obtaining a third probability corresponding to each of the candidate tokens; If the third probability is greater than the second probability, the corresponding candidate token is used as the result of the first model; otherwise, the corresponding candidate token is discarded and resampled until a complete inference result is generated.

6. A reasoning method based on a large language model, characterized in that: The inference method uses collaborative inference between a cloud and a mobile terminal; a first model is deployed on the cloud, and a second model is deployed on the mobile terminal; wherein the number of parameters of the first model is greater than the number of parameters of the second model; the method includes: The mobile terminal sends the received inference request to the cloud; The cloud generates a key-value cache corresponding to the inference request based on the first model; The cloud quantizes the key-value cache to obtain a first quantized key-value cache, and sends the first quantized key-value cache to the mobile terminal; The mobile terminal generates a candidate token sequence corresponding to the inference request based on the first quantized key-value cache and the second model, and sends the candidate token sequence to the cloud; The cloud uses the key-value cache and the first model to verify the candidate token sequence to obtain an inference result.

7. The method according to any one of claims 1 to 6, characterized in that: The first model and the second model are compatible.

8. The method according to any one of claims 1 to 6, characterized in that: Before transmitting data, the cloud and the mobile terminal encrypt the data to be transmitted.

9. An electronic device, characterized in that: include: processor, memory and bus, wherein, The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, enable the computer to perform the method according to any one of claims 1 to 5.

11. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 5 is executed.

12. A reasoning system based on a large language model, characterized in that: It includes a cloud and a mobile terminal; wherein the cloud is deployed with a first model, and the mobile terminal is deployed with a second model; wherein the parameter amount of the first model is greater than the parameter amount of the second model; the mobile terminal can execute the method described in any one of claims 1 to 3, and the cloud is capable of executing the method described in claim 4 or 5.

Images