Optimization method of end-side LLM in end-cloud LLM hybrid service framework

Abstract

An optimization method for end-side LLM under an end-cloud LLM hybrid service framework collects historical interaction data between users and cloud-side LLM through a terminal and builds an external database, evaluates the value of each element therein and generates an optimal database subset; when receiving user input, the terminal weights the generation result of the end-side LLM and the k-nearest neighbor search result in the optimal database subset to obtain the end-side reply result, thereby achieving reasoning enhancement. The present invention significantly improves the generation quality of end-side LLM by using historical interaction records between users and the cloud side to establish an external database, and compresses the database through offline and online subset selection algorithms, both of which have an approximation ratio guarantee close to (1-1 / e), and the perplexity (PPL) brought about is reduced more than other database compression algorithms.

Description

Technical Field

[0001] The present invention relates to a technology in the field of neural network applications, specifically an optimization method for end-side LLM in an end-cloud LLM hybrid service framework. Background Art

[0002] In order to support the deployment of large language models (LLMs) in various application scenarios with different computing resources, many companies have launched a series of LLMs with different parameter sizes. In this series of LLMs, the largest models usually have a parameter scale of tens of billions, are deployed in the resource-rich cloud, and provide services to a large number of users through online API requests. The parameter scale of lightweight models is usually billions, which supports deployment on resource-constrained mobile devices to provide local real-time services to users. The deployment of cloud-side LLM and terminal-side lightweight LLM jointly ensures that users can enjoy LLM services at any time. Specifically, when the cloud-side LLM service is available to users, the terminal directly sends a request and the cloud returns a response; but when the cloud-side LLM service is unavailable, the terminal-side LLM needs to be awakened for local services. Summary of the invention

[0003] In view of the shortcomings of the existing technology that cannot effectively build a high-quality knowledge database and cannot guarantee low local inference latency, the present invention proposes an optimization method for end-side LLM under the end-cloud LLM hybrid service framework. The external database is established by using the historical interaction records of users and the cloud side to significantly improve the quality of end-side LLM generation, and the database is compressed through offline and online subset selection algorithms. Both have an approximation ratio guarantee close to (1-1 / e), and the perplexity level (PPL) is reduced more than other database compression algorithms.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to an optimization method for end-side LLM under an end-cloud LLM hybrid service framework. After collecting historical interaction data between users and cloud-side LLM through a terminal and constructing an external database, the value of each element therein is evaluated and an optimal database subset is generated. When receiving user input, the terminal weights the generation result of the end-side LLM and the k-nearest neighbor retrieval result in the optimal database subset to obtain an end-side reply result, thereby realizing reasoning enhancement.

[0006] The external database preprocesses a new conversation into a token sequence (w1, w2, ..., w l), the token sequence is divided into (l-1) context-target pairs; the context in each context-target pair is encoded into a dense vector through the end-side LLM, and is combined with its corresponding target to form a key-value pair; finally, all key-value pairs collected during the historical period are combined to form a complete external database.

[0007] The optimal database subset is achieved by any of the following methods:

[0008] a) Offline database selection algorithm, after randomly selecting a subset R of size B from the complete database except the current database subset, select the key-value pairs corresponding to the b context-target pairs with the smallest generation probability from the subset R, add them to the database subset and repeat until the size of the database subset reaches the database upper limit L;

[0009] b) Online database selection algorithm, which is incrementally updated online when generating key-value pair data streams. Specifically, when a new key-value pair is generated, its corresponding generation probability p(t|c,W,S j ), and generate the b with the smallest probability among the B new key-value pairs collected j key-value pairs are added to the candidate subset S j , repeat this operation until the size of the candidate subset reaches the database upper limit L.

[0010] The generation probability p(t|c,W,U)=(1-α)p DLM (t|c,W)+αp knn (t|c,U), where: p DLM (t|c,W) represents the probability of context c generating target t in the client-side LLM, p knn (t|c,U) represents the generation probability obtained by performing k-nearest neighbor search on context c in the external database U, which is implemented using but not limited to the method described in "Generalization through Memorization: Nearest Neighbor Language Models".

[0011] The end-side response result refers to: for the input token sequence The predicted distribution for the next token is

[0012] Technical Effects

[0013] The present invention uses the historical interaction records between users and the cloud side to establish an external database to enhance the quality of the terminal side generation, and uses the marginal generation probability p(t|c,W,S j ) method to determine the key-value pair with the greatest value for the current database subset, and use it as the subset selection method. Compared with the prior art, the present invention uses an external database established by historical interaction records between users and the cloud side, combined with the k-nearest neighbor retrieval method, which can significantly reduce the perplexity of the end-side reasoning; using the marginal generation probability p(t|c,W,S j ) can effectively compress the database, and the perplexity reduction it brings is greater than that of other database compression algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of cloud-side LLM and device-side LLM providing services to users in different situations;

[0015] Figure 2 Schematic diagram of using the cloud-side LLM to build a complete database, compress the database using the database subset selection algorithm, and enhance the end-side LLM using the database subset;

[0016] Figure 3 Schematic diagram of compressing a complete database using an offline subset selection algorithm;

[0017] Figure 4 Schematic diagram of compressing a complete database using an online subset selection algorithm;

[0018] Figure 5 and Figure 6 It is a schematic diagram of the effect of the embodiment. DETAILED DESCRIPTION

[0019] like Figure 1 As shown, this embodiment relates to a method for enhancing the end-side LLM under the end-cloud LLM hybrid service framework, including:

[0020] Step 1: Build and initialize the external database: Filter out 5 datasets from the ShareGPT dataset by topic. Each topic-related dataset simulates a user's local dataset. For each user dataset, select about 70% of the data to build a complete external database, and the remaining 30% of the data is used for testing.

[0021] Step 2: Generate the optimal subset: preprocess the user training data or the user conversation data read in sequence into the form of context-target pairs, and further encode the context using Qwen-1.8B. After the encoded vector and the corresponding target form a key-value pair, apply the offline or online subset selection algorithm to the set of all key-value pairs and return the optimal subset.

[0022] Step 3: End-side LLM enhancement based on the optimal subset of the database: The optimal subset obtained in the second step is used as the external database. For the autoregressive task during on-end inference, the generation result of the end-side LLM and the weighted k-nearest neighbor retrieval result in the external database are used to predict the next word when generating the next word.

[0023] The prediction is achieved by using but not limited to the method described in "Generalization through Memorization: Nearest Neighbor Language Models".

[0024] After specific actual experiments, the experiment was conducted on the topic-screened ShareGPT dialogue dataset. There are 5 user simulation datasets in total. By comparing the PPL of Qwen-1.8B without external database enhancement and Qwen-1.8B enhanced with a database subset on the test set, the results are shown in Table 1. The experimental results show that using offline and online selected database subsets for enhancement can significantly reduce PPL compared to simply using end-side LLM.

[0025] Table 1: Confusion records of different users using database subsets to enhance the client-side LLM

[0026]

[0027] like Figure 5 As shown, the full database is used for comparison with only the database containing 10 5 We investigated the enhanced inference latency of a subset of a database of key-value pairs and found that compressing the database can effectively reduce the inference latency by up to 2.3×.

[0028] like Figure 6 As shown in the figure, the changes in PPL and inference latency under different database subset size limits are compared. It is found that increasing the database subset size can improve the generation quality, while reducing the database subset size can improve the generation speed. In actual scenarios, dynamic selection can be made according to needs.

[0029] Compared with the prior art, the present invention uses the historical interactions between users and cloud-side LLM to build an external database, which can effectively enhance the quality of the results generated on the end side. At the same time, it uses an offline / online database subset selection algorithm to effectively compress the database to improve the end-side reasoning speed.

[0030] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for enhancing the end-side LLM in an end-cloud LLM hybrid service framework, characterized in that: After collecting historical interaction data between users and cloud-side LLM through the terminal and building an external database, the value of each element is evaluated and the optimal database subset is generated. When receiving user input, the terminal weights the generation result of the terminal-side LLM and the k-nearest neighbor search result in the optimal database subset to obtain the terminal-side reply result, realizing reasoning enhancement. The optimal database subset is achieved by any of the following methods: a) Offline database selection algorithm, after randomly selecting a subset R of size B from the complete database except the current database subset, select the key-value pairs corresponding to the b context-target pairs with the smallest generation probability from the subset R, add them to the database subset and repeat until the size of the database subset reaches the database upper limit L; b) Online database selection algorithm, which is incrementally updated online when generating key-value pair data streams. Specifically, when a new key-value pair is generated, its corresponding generation probability p(t|c,W,S j ), and generate the b with the smallest probability among the B new key-value pairs collected j key-value pairs are added to the candidate subset S j , repeat this operation until the candidate subset size reaches the database upper limit L.

2. The method for enhancing the end-side LLM in the end-cloud LLM hybrid service framework according to claim 1 is characterized in that: The external database preprocesses a new conversation into a token sequence (w1, w2, ..., w l ), the token sequence is split into (l-1) context-target pairs; The context in each context-target pair is then encoded into a dense vector through the client-side LLM and combined with its corresponding target to form a key-value pair. Finally, all key-value pairs collected during the historical period are combined to form a complete external database.

3. The method for enhancing the end-side LLM in the end-cloud LLM hybrid service framework according to claim 1 is characterized in that: The generation probability , where: p DLM (t|c,W) represents the probability of context c generating target t in the client-side LLM, p knn (t|c,U) represents the generation probability of context c obtained by performing j-nearest neighbor search in the external database U.

4. The method for enhancing the end-side LLM in the end-cloud LLM hybrid service framework according to claim 1 is characterized in that: The end-side response result refers to: for the input token sequence The predicted distribution for the next token is 5. The method for enhancing the end-side LLM in the end-cloud LLM hybrid service framework according to any one of claims 1 to 4 is characterized in that: include: Step 1: Build and initialize the external database: Select 5 datasets from the ShareGPT dataset by topic. Each topic-related dataset simulates a user's local dataset. For each user dataset, select 70% of the data to build a complete external database, and the remaining 30% of the data is used for testing. Step 2: Generate the optimal subset: pre-process the user training data or the user conversation data read in sequence into the form of context-target pairs, and further encode the context using Qwen-1.8B. After the encoded vector and the corresponding target form a key-value pair, apply the offline or online subset selection algorithm to the set of all key-value pairs and return the optimal subset; Step 3: End-side LLM enhancement based on the optimal subset of the database: The optimal subset obtained in the second step is used as the external database. For the autoregressive task during on-end inference, the generation result of the end-side LLM and the weighted k-nearest neighbor retrieval result in the external database are used to predict the next word when generating the next word.

Images