A lightweight large model running method based on end-side deployment

Abstract

The application provides a lightweight large model running method based on end-side deployment, comprising: obtaining device running data and storing a model file in a mixed precision format according to quantization precision supported by a terminal device; determining a mixed precision quantization strategy and performing strategy analysis on the device running data; reserving a Key-Value cache file obtained through model inference in a historical dialogue; calculating a semantic embedding vector of a second user request, and calculating a cosine similarity of the semantic embedding vector and a first V vector, and determining whether the second user request has a similar intention according to a calculation result; if yes, performing incremental inference on the second user request by reusing the Key-Value cache file. The application dynamically adjusts a quantization level by monitoring a terminal device state in real time, reuses cache data of semantically related requests, and combines batch request inferences, so that the terminal device energy consumption and performance are balanced, repeated calculation is reduced, and the device use time and user experience are improved.

Description

Technical Field

[0001] This invention relates to the field of large model invocation technology, specifically to a lightweight large model running method based on edge deployment. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence (AI) technology, front-end applications have become increasingly reliant on cloud services for data processing and model inference. This has led to issues such as network dependency, response latency, high server costs, and risks of user privacy breaches. Furthermore, for large-scale super-applications with massive models, the simultaneous online presence of tens of millions of users could become prohibitively expensive due to the high cost of cloud computing power alone. Therefore, a new concept—On-Device AI—was introduced at the end of 2023. This allows AI models to perform inference directly on terminal devices such as smartphones and IoT devices without connecting to servers or the cloud. This effectively protects user privacy, reduces network latency and computing costs, and allows users to access the technology anytime, anywhere. It offers low latency, enhanced security, and greater flexibility, representing a significant technological trend in the field of AI.

[0003] However, cloud computing power can be increased by adding hardware resources such as servers and GPUs, but edge products do not have this capability and need to maintain performance under extremely limited hardware resources. Currently, they face the following severe challenges:

[0004] 1. High power consumption and heat generation of terminal devices. Large Language Model (LLM) inference is a computationally intensive task. Running at full load for a long time will cause the terminal device's battery to be quickly depleted, and the increased body temperature will trigger system-level throttling (CPU Throttling), which will cause inference to lag.

[0005] 2. Inflexible Model Quantization Strategy: Existing terminal devices typically employ static quantization, which converts model parameters to INT4 or INT8 formats and fixes this format for the entire inference process. While using INT4 format significantly reduces hardware resource consumption, it results in substantial accuracy loss. INT8 format, on the other hand, significantly accelerates inference but requires considerable hardware resources. Therefore, static quantization strategies cannot effectively balance efficiency and accuracy. For example, they cannot achieve high performance when battery power is sufficient, and cannot further degrade to maintain basic service when battery power is low.

[0006] 3. Wasted resources due to redundant calculations: Multi-turn dialogue systems continuously accumulate historical interaction content and generate contextual information to ensure that subsequent responses are based on previous content, thus avoiding contradictions or repetitions and maintaining semantic coherence. However, this contextual information contains a large amount of repetitive content, and existing inference engines often recalculate the Attention matrix for each round of input. This significantly increases token consumption and causes inference latency, further increasing the resource consumption of the terminal device.

[0007] Therefore, with the deployment of edge AI models, there is an urgent need for a lightweight method for running large models that can balance the energy consumption and performance of terminal devices. Summary of the Invention

[0008] This invention provides a lightweight large model operation method based on edge deployment, which is mainly used to solve the problems of high power consumption and heat generation of terminal devices, inflexible model quantization strategies, and resource waste caused by repeated calculations in existing edge AI solutions. By dynamically adjusting the quantization level in real time by monitoring the terminal device status, reusing cached data of semantically related requests, and merging batch request inference, it achieves the effect of balancing terminal device power consumption and performance, reducing repeated calculations, and improving device usage time and user experience.

[0009] The present invention achieves the above objectives through the following technical solutions:

[0010] A lightweight method for running large models based on edge deployment includes:

[0011] S1: Obtain device operation data by monitoring the running status of the terminal device during the large model runtime in real time, and store the model file in a mixed precision format according to the quantization precision supported by the terminal device.

[0012] S2: Determine a mixed-precision quantization strategy by taking the performance and power consumption of the terminal device as constraints, perform strategy analysis on the device operation data and generate an adaptive quantization strategy according to the mixed-precision format, and execute the adaptive quantization strategy.

[0013] S3: Identify the first user request with continuous semantic relevance in a multi-turn dialogue, and use KV Cache to retain the Key-Value cache file obtained by model inference of the first user request in the historical dialogue.

[0014] The Key-Value cache file includes a first K vector and a first V vector corresponding to the N tokens generated by model inference.

[0015] S4: When a new second user request is received in the multi-turn dialogue, obtain the semantic embedding vector of the second user request, calculate the cosine similarity between the semantic embedding vector and the first V vector in the Key-Value cache file, and determine whether the second user request has a similar intent based on the calculation result.

[0016] S5: If there is a similar intent, then perform incremental reasoning on the second user request by reusing the Key-Value cache file.

[0017] According to the present invention, a lightweight large model running method based on edge deployment is provided, wherein the quantization accuracy supported by the terminal device includes at least a first accuracy and a second accuracy, wherein the first accuracy is less than the second accuracy; the model file setting steps include:

[0018] S11: Calculate the quantization residual of the large model at the second precision using the first precision as the baseline parameter.

[0019] S12: Obtain the model baseline weights stored with the first precision and the model residual weights stored with the quantized residuals. The model file stores the model baseline weights and model residual weights to form a model weight matrix.

[0020] According to the present invention, a lightweight large-scale model operation method based on edge deployment is provided. The mixed-precision quantization strategy includes a high-load mode, a balanced mode, and a low-power mode. The mixed-precision quantization strategy mode is selected for the device operation data according to the constraints. The quantization strategies under different modes include:

[0021] When the high-load mode is selected, the model weights are reconstructed by superimposing the model baseline weights and the model residual weights, and the activation value of the first model weights is obtained. The model weights are then loaded for model quantization.

[0022] When the balance mode is selected, the model baseline weights are used as the second model weights and the activation values ​​of the second model weights are obtained. The second model weights are then loaded for model quantization.

[0023] When the low-power mode is selected, after model preprocessing, the model baseline weights are used as the third model weights and the activation values ​​of the third model weights are obtained. The third model weights are then loaded for model quantization. The model preprocessing is either model pruning or adding an early termination mechanism.

[0024] According to the present invention, a lightweight large model running method based on edge deployment is provided. In the balanced mode, the SIMD instruction set is used to accelerate model quantization. The steps include:

[0025] S211: Select the appropriate SIMD instruction set according to the hardware platform of the terminal device. The SIMD instruction set includes at least SIMD loading instructions, SIMD unpacking instructions, SIMD multiplication instructions, and SIMD accumulation instructions.

[0026] S212: Pack multiple second model weights and their activation values ​​into a single package, and load the packaged data into the SIMD register at once using the SIMD loading instruction.

[0027] S213: The packaged data is decompressed into register data with the second precision data type by the SIMD unpacking instruction.

[0028] S214: Perform parallel operations on the register data using SIMD multiplication instructions, perform SIMD multiplication on the multiple product results obtained from the parallel operations and their corresponding scaling factors, and then output the accumulated results using the SIMD accumulation instructions.

[0029] According to the present invention, a lightweight large model running method based on edge deployment includes the following model pruning steps:

[0030] S221: Determine the pruning point, and add an auxiliary classification head after the network layer at the corresponding position in the model according to the pruning point.

[0031] S222: After the data stream flows through the network layer, the probability distribution is directly output through the auxiliary classification head and the process jumps to the last layer for model calculation.

[0032] The steps for adding the early departure mechanism include:

[0033] S231: Add an early termination module after each network layer, the early termination module being used to calculate the early termination score of the network layer.

[0034] S232: Evaluate the output of the network layer based on the early termination score. If early termination is decided, output the result of the current network layer in advance; otherwise, continue to propagate to the next network layer for calculation, and repeat this step until the output result is output.

[0035] According to the lightweight large model running method based on edge deployment provided by the present invention, the identification step of the first user request in step S3 includes:

[0036] S31: Store the historical semantic fingerprints of the multi-turn dialogue by maintaining an LRU cache pool.

[0037] S32: Extract the sentence vector of the first user request, and perform dimensionality reduction and hash encoding on the sentence vector to generate a fixed-length semantic fingerprint.

[0038] S33: Set a semantic similarity threshold, calculate the Hamming distance between the semantic fingerprint and the historical semantic fingerprint, and if the Hamming distance is less than the similarity threshold, then determine the first user request as a semantically similar request.

[0039] According to the lightweight large model running method based on edge deployment provided by the present invention, the method for calculating cosine similarity in step S4 includes:

[0040] S41: The LRU cache pool also stores the Key-Value cache file, and the second user request is converted into the semantic embedding vector by the Embedding model.

[0041] S42: Calculate the cosine similarity between the semantic embedding vector and the first V vector in the Key-Value cache file using a similarity function; wherein, the similarity function is:

[0042]

[0043] in, The semantic embedding vector of the second user request. The first V vector in the Key-Value cache file.

[0044] S43: Set the similarity threshold to 0.8, when If so, it is determined that the second user request has a similar intent.

[0045] According to the lightweight large model running method based on edge deployment provided by the present invention, the incremental inference process in step S5 includes:

[0046] S51: The large model generates output token by token according to the second user request, performs a linear transformation on the current token to obtain the second Q vector, the second K vector, and the second V vector; and reads all the first K vectors and first V vectors in the Key-Value cache file, with all the first K vectors forming the K matrix and all the first V vectors forming the V matrix.

[0047] S52: Calculate the attention score based on the second Q vector and the K matrix. After normalization, the attention score is weighted and summed on the V matrix to obtain the context vector.

[0048] S53: Input the context vector into the FFN layer of the large model, calculate and output the logits vector, and convert the logits vector into a probability distribution.

[0049] S54: Sample the probability distribution to obtain a new token, and append the second K vector and the second V vector to the Key-Value cache file, incrementing the newly generated token sequence by one.

[0050] Repeat steps S51 to S54 until the newly obtained token is the predefined end marker.

[0051] According to the present invention, a lightweight large-model running method based on edge deployment is provided, which uses dynamic batch processing to preprocess user requests, and performs concurrent inference on multiple user requests in the same batch within a short period of time according to steps S1 to S5. The dynamic batch processing steps include:

[0052] S6: Set a time window, merge the user requests received within the time window into a dynamic batch, and perform consistency processing on the length of the sequence in the dynamic batch.

[0053] According to the lightweight large model running method based on edge deployment provided by the present invention, the length consistency processing step of the sequence in the dynamic batch in step S6 includes:

[0054] S61: Set a length threshold and determine whether the length of the sequence in the dynamic batch exceeds the length threshold. If so, truncate the excess part.

[0055] S62: Find the longest sequence in the dynamic batch and dynamically fill the relatively short sequences according to the length of the longest sequence.

[0056] S63: Configure Attention Mask, which identifies the valid bits and padding bits in the sequence, and is used to mask the padding bits during model calculation.

[0057] Therefore, the present invention has the following beneficial effects:

[0058] 1. This invention dynamically adjusts the quantization level by monitoring the terminal device status in real time, reuses cached data from semantically related requests, and merges batch request inference. This reduces the terminal device's inference power consumption and keeps the device temperature under control, ensuring that the core functions of the model remain available even in low-power mode. Compared to traditional solutions, this invention effectively solves the power consumption problem of running lightweight large models deployed on the terminal side, significantly improving user experience and satisfaction, and providing key technical support for the terminal-side deployment of lightweight large models.

[0059] 2. This invention determines a hybrid precision quantization strategy by using the performance and power consumption of the terminal device as constraints, so that the model quantization precision can dynamically adapt to the terminal device state, thereby achieving a good balance between efficiency and precision. For example, it can achieve high performance when the battery is full, and still maintain the availability of the model's core functions in low power mode.

[0060] 3. This invention utilizes the SIMD instruction set to process multiple quantized data in parallel, which can significantly improve the computational efficiency of low-precision inference, reduce latency, and further reduce resource consumption.

[0061] 4. In the optimization of multi-turn dialogue based on KV Cache, this invention uses cosine similarity to determine the semantic similarity of new user requests. It only requires calculating the dot product and vector magnitude, and the formula is simple and computationally efficient. Moreover, cosine similarity measures the cosine of the angle between two vectors in space, so it only focuses on the direction of the vectors rather than their absolute size, thereby avoiding the interference of word frequency or intensity correlation in the sentence when semantic judgment is caused by embedding vector length calculation.

[0062] 5. This invention employs dynamic batch processing to preprocess user requests, merging sparse or concurrently arriving requests on the terminal device within a small time window. Padding and Masking are used to consistently extend the request length and mask padding bits, respectively. This facilitates the conversion of merged requests into regular tensors. The merged tensors are then input into the terminal device's GPU, utilizing the GPU's parallel computing capabilities to derive the results of all requests at once. Compared to serial execution, this significantly improves system resource utilization, and the increased throughput reduces latency, thereby enhancing the user experience.

[0063] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0064] Figure 1 This is a flowchart of an embodiment of the present invention of a lightweight large model operation method based on edge deployment.

[0065] Figure 2 This is a block diagram of the application system hardware architecture in an embodiment of a lightweight large model running method based on edge deployment according to the present invention.

[0066] Figure 3 This is a flowchart of a method for hybrid precision quantization strategy analysis in an embodiment of a lightweight large model running method based on edge deployment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0068] An Implementation Example of a Lightweight Large Model Running Method Based on Edge Deployment

[0069] See Figure 1 The present invention relates to a lightweight large-scale model running method based on edge deployment, comprising:

[0070] S1: Obtain device operation data by monitoring the running status of the terminal device during the large model runtime in real time, and store the model file in a mixed precision format according to the quantization precision supported by the terminal device.

[0071] S2: Determine a mixed-precision quantization strategy by taking the performance and power consumption of the terminal device as constraints, perform strategy analysis on the device operation data and generate an adaptive quantization strategy according to the mixed-precision format, and execute the adaptive quantization strategy.

[0072] S3: Identify the first user request with continuous semantic relevance in a multi-turn dialogue, and use KV Cache to retain the Key-Value cache file obtained by model inference of the first user request in the historical dialogue.

[0073] The Key-Value cache file includes a first K vector and a first V vector corresponding to the N tokens generated by model inference.

[0074] S4: When a new second user request is received in the multi-turn dialogue, obtain the semantic embedding vector of the second user request, calculate the cosine similarity between the semantic embedding vector and the first V vector in the Key-Value cache file, and determine whether the second user request has a similar intent based on the calculation result.

[0075] S5: If there is a similar intent, then perform incremental reasoning on the second user request by reusing the Key-Value cache file.

[0076] Specifically, this embodiment presets a monitoring period, and polls the status parameters of the terminal device according to the monitoring period during the large model's operation. The specific monitoring process is illustrated using Android monitoring for one monitoring period as an example:

[0077] S101: Register a BroadcastReceiver and listen for ACTION_BATTERY_CHANGED (sticky broadcast) to parse the current battery percentage and charging status.

[0078] S102: Reads the system's predefined thermal zone (temperature sensing) file to obtain the raw temperature sensor values ​​for the processor or battery.

[0079] S103: Reads the system information file and calculates the real-time utilization of CPU and memory, including: calculating the instantaneous utilization based on the difference between user mode, kernel mode, idle mode, etc., and calculating the memory utilization and available memory based on fields such as MemTotal (total physical memory of the system) and MemAvailable (memory size at the application level).

[0080] S104: The above monitoring data is filtered using a moving average filtering algorithm, including: maintaining a data window of a fixed size, such as the most recent 3 monitoring data, and using the arithmetic mean of the monitoring data within the data window as the valid output of the equipment operation data.

[0081] See Figure 2 Specifically, the hardware system of this embodiment, which applies a lightweight large-model running method based on edge deployment, includes a bottom layer 10, a middle layer 20, and a top layer 30. The bottom layer 10 is a heterogeneous computing execution layer, which efficiently executes model inference tasks optimized by the upper layer through the collaborative work of CPU, GPU, and dedicated NPU. The middle layer 20 is an intelligent management and scheduling layer, serving as the system control center. It includes a scheduler, quantizer, and memory manager. The scheduler determines the running mode based on the state, the quantizer dynamically switches model precision, and the memory manager optimizes KV cache memory usage. The top layer 30 is a sensing and input layer, which collects user requests and real-time device status data through application call interfaces, enabling the large model to dynamically adjust its running strategy based on the real-time resource status of the edge device.

[0082] In this embodiment, the quantization precision supported by the terminal device includes at least a first precision and a second precision, wherein the first precision is less than the second precision; the model file setting steps include:

[0083] S11: Calculate the quantization residual of the large model at the second precision using the first precision as the baseline parameter.

[0084] S12: Obtain the model baseline weights stored with the first precision and the model residual weights stored with the quantized residuals. The model file stores the model baseline weights and model residual weights to form a model weight matrix.

[0085] Specifically, in this embodiment, the first precision is INT4 and the second precision is INT8. The calculation process of the quantization residual includes:

[0086] S111: Quantize the original floating-point weight matrix W_float by channel or group: Calculate the INT4 quantization parameters, including the scaling factor scale_int4 and the zero point zero_point_int4, and obtain the INT4 integer weights, including W_int4=quantize_int4(W_float,scale_int4,zero_point_int4).

[0087] S112: After adjusting the quantization precision to INT8, repeat step S111 to obtain the INT8 quantization parameters: scale_int8, zero_point_int8, and INT8 integer weights: W_int8=quantize_int8(W_float,scale_int8,zero_point_int8).

[0088] S113: Map the INT4 weights W_int4 and INT8 weights W_int8 to the same numerical scale to unify the numerical space. For example, the INT4 weights can be dequantized to the INT8 scale: W_dequant=(W_int4-zero_point_int4)*(scale_int4 / scale_int8)+zero_point_int8.

[0089] S114: Calculate the difference between the INT8 weight W_int8 and the dequantized weight W_dequant: Residual_float=W_int8-W_dequant, and quantize this difference to obtain the quantization residual.

[0090] Specifically, the quantization precision mentioned above in this embodiment is only exemplary and not the only method. Other methods include FP16, FP32, FP64, etc., which can be determined according to the actual situation of the terminal device.

[0091] In this embodiment, the mixed-precision quantization strategy includes a high-load mode, a balanced mode, and a low-power mode. The mixed-precision quantization strategy mode is selected based on the constraints on the device operating data. The quantization strategies under different modes include:

[0092] When the high-load mode is selected, the model weights are reconstructed by superimposing the model baseline weights and the model residual weights, and the activation value of the first model weights is obtained. The model weights are then loaded for model quantization.

[0093] When the balance mode is selected, the model baseline weights are used as the second model weights and the activation values ​​of the second model weights are obtained. The second model weights are then loaded for model quantization.

[0094] When the low-power mode is selected, after model preprocessing, the model baseline weights are used as the third model weights and the activation values ​​of the third model weights are obtained. The third model weights are then loaded for model quantization. The model preprocessing is either model pruning or adding an early termination mechanism.

[0095] See Figure 3 Specifically, in this embodiment, when the strategy determines that the target mode is different from the current mode, a switching is triggered. Taking power and temperature-related device status data, with the first precision being INT4 and the second precision being INT8 as an example, the mode switching is explained as follows:

[0096] When the battery level is >50% and the temperature is <40℃, switch to high-load mode to ensure inference speed ≥15 Tokens / s. The policy controller sends instructions to the inference engine, and the engine executes the following steps:

[0097] S201-1: For the layer dynamic loading model residual weights to be calculated, the model baseline weights and model residual weights are reconstructed into complete first model weights in memory.

[0098] S201-2: Switch the computing kernel from the 4-bit kernel to the Int8 kernel and set the execution path to execute_full_int8().

[0099] When the battery level is less than 20% or the temperature is ≥45℃, the system switches to low-power mode, enabling a simplified inference head by retaining only core inference capabilities. In this mode, power consumption is reduced by another 30% compared to balanced mode. The policy controller sends instructions to the inference engine, which then executes the following steps:

[0100] S202-1: Use model baseline weights and switch to a 4-bit kernel.

[0101] S202-2: Switch the execution path from execute_full_int4() to execute_early_exit_int4(exit_layer=6), where exit_layer=6 means only the first 6 layers are retained for model inference so that the output can be performed in advance.

[0102] When the battery level is between 20% and 50% or the temperature is between 40°C and 45°C, switch to Balanced Mode to ensure inference speed ≥10 Tokens / s and power consumption reduced by 40%. The policy controller sends instructions to the inference engine, and the engine executes the following steps:

[0103] S203-1: Switch the execution path from early exit subgraph back to execute_full_int4().

[0104] S203-2: Uses model baseline weights and switches to a 4-bit kernel, and utilizes the SIMD instruction set to accelerate model quantization.

[0105] In this embodiment, in the balanced mode, the SIMD instruction set is used to accelerate model quantization, and the steps include:

[0106] S211: Select the appropriate SIMD instruction set according to the hardware platform of the terminal device. The SIMD instruction set includes at least SIMD loading instructions, SIMD unpacking instructions, SIMD multiplication instructions, and SIMD accumulation instructions.

[0107] S212: Pack multiple second model weights and their activation values ​​into a single package, and load the packaged data into the SIMD register at once using the SIMD loading instruction.

[0108] S213: The packaged data is decompressed into register data with the second precision data type by the SIMD unpacking instruction.

[0109] S214: Perform parallel operations on the register data using SIMD multiplication instructions, perform SIMD multiplication on the multiple product results obtained from the parallel operations and their corresponding scaling factors, and then output the accumulated results using the SIMD accumulation instructions.

[0110] Specifically, in this embodiment, the width of the SIMD register is large enough to enable parallel data processing. For example, a 128-bit wide SIMD register can simultaneously hold 32 4-bit integers or 16 8-bit numbers. Utilizing the SIMD instruction set to process multiple quantized data in parallel can significantly improve the computational efficiency of low-precision inference, such as INT8 and INT4, while reducing latency and resource consumption.

[0111] In this embodiment, the model pruning step includes:

[0112] S221: Determine the pruning point, and add an auxiliary classification head after the network layer at the corresponding position in the model according to the pruning point.

[0113] S222: After the data stream flows through the network layer, the probability distribution is directly output through the auxiliary classification head and the process jumps to the last layer for model calculation.

[0114] Specifically, the standard Transformer model in this embodiment typically has 12-32 Decoder Blocks. During the training phase, an Auxiliary Head is added after the 6th layer.

[0115] The steps for adding the early departure mechanism include:

[0116] S231: Add an early termination module after each network layer, the early termination module being used to calculate the early termination score of the network layer.

[0117] S232: Evaluate the output of the network layer based on the early termination score. If early termination is decided, output the result of the current network layer in advance; otherwise, continue to propagate to the next network layer for calculation, and repeat this step until the output result is output.

[0118] In this embodiment, the identification step of the first user request in step S3 includes:

[0119] S31: Store the historical semantic fingerprints of the multi-turn dialogue by maintaining an LRU cache pool.

[0120] S32: Extract the sentence vector of the first user request, and perform dimensionality reduction and hash encoding on the sentence vector to generate a fixed-length semantic fingerprint.

[0121] S33: Set a semantic similarity threshold, calculate the Hamming distance between the semantic fingerprint and the historical semantic fingerprint, and if the Hamming distance is less than the similarity threshold, then determine the first user request as a semantically similar request.

[0122] In this embodiment, the method for calculating cosine similarity in step S4 includes:

[0123] S41: The LRU cache pool also stores the Key-Value cache file, and the second user request is converted into the semantic embedding vector by the Embedding model.

[0124] S42: Calculate the cosine similarity between the semantic embedding vector and the first V vector in the Key-Value cache file using a similarity function; wherein, the similarity function is:

[0125]

[0126] in, The semantic embedding vector of the second user request. The first V vector in the Key-Value cache file.

[0127] S43: Set the similarity threshold to 0.8, when If so, it is determined that the second user request has a similar intent.

[0128] Specifically, this embodiment determines whether a new user request has a similar intent by calculating cosine similarity. The threshold is .when At this time, the vector and The angle between The vectors have completely identical directions and semantics; when At this time, the vector and The angle between Vectors with highly similar directions and semantic similarity; when At this time, the vector and The angle between Vectors are perpendicular and semantically independent; when At this time, the vector and The angle between The vectors have completely opposite directions and opposite meanings. The above similarity thresholds are only preferred values, not the only ones to choose from, and can be selected according to actual needs. Select from the range.

[0129] Specifically, in this embodiment, the cosine similarity is used to determine the semantic similarity of new user requests in the multi-turn dialogue optimization based on KV Cache. Only the dot product and vector magnitude need to be calculated, and the formula is simple and computationally efficient. Moreover, the cosine similarity measures the cosine value of the angle between two vectors in space. Therefore, it only focuses on the direction of the vector rather than its absolute size, so as to avoid the interference that the length of the sentence embedding vector is related to the word frequency or intensity in the sentence but not to the semantics.

[0130] In this embodiment, the incremental inference process in step S5 includes:

[0131] S51: The large model generates output token by token according to the second user request, performs a linear transformation on the current token to obtain the second Q vector, the second K vector, and the second V vector; and reads all the first K vectors and first V vectors in the Key-Value cache file, with all the first K vectors forming the K matrix and all the first V vectors forming the V matrix.

[0132] S52: Calculate the attention score based on the second Q vector and the K matrix. After normalization, the attention score is weighted and summed on the V matrix to obtain the context vector.

[0133] S53: Input the context vector into the FFN layer of the large model, calculate and output the logits vector, and convert the logits vector into a probability distribution.

[0134] S54: Sample the probability distribution to obtain a new token, and append the second K vector and the second V vector to the Key-Value cache file, incrementing the newly generated token sequence by one.

[0135] Repeat steps S51 to S54 until the newly obtained token is the predefined end marker.

[0136] Specifically, in this embodiment, the attention score is calculated by performing a dot product of the second Q vector with all the first K vectors.

[0137] Specifically, in this embodiment, the FFN layer is located after the self-attention layer in the Transformer model architecture. The context vector output by the attention layer encodes all relevant information that should be considered at the current generation position. The FFN layer is used to perform non-linear transformations and dimensionality increases / decreases on the context vector and output it. After passing through a linear projection layer, the output is converted to a logits vector. The length of the logits vector is equal to the vocabulary size, which includes several candidate tokens.

[0138] Specifically, in this embodiment, the Softmax function is used to convert the logits vector into a probability distribution, that is, the probability that each token in the vocabulary will become the new token.

[0139] Specifically, this embodiment employs Top-k / Top-p sampling, sampling only from the k tokens with the highest probabilities or from the set of candidate tokens with a cumulative probability of p, to eliminate low-probability noise. Here, p is the cumulative probability, calculated by sorting all tokens in the vocabulary from highest to lowest probability and accumulating the probability from the highest probability. The accumulated tokens constitute the candidate token set.

[0140] In this embodiment, dynamic batch processing is used to preprocess user requests. According to steps S1-S5, multiple user requests in the same batch within a short period are concurrently inferred. This dynamic batch processing step includes:

[0141] S6: Set a time window, merge the user requests received within the time window into a dynamic batch, and perform consistency processing on the length of the sequence in the dynamic batch.

[0142] In this embodiment, the length consistency processing step of the sequence in the dynamic batch in step S6 includes:

[0143] S61: Set a length threshold and determine whether the length of the sequence in the dynamic batch exceeds the length threshold. If so, truncate the excess part.

[0144] S62: Find the longest sequence in the dynamic batch and dynamically fill the relatively short sequences according to the length of the longest sequence.

[0145] S63: Configure Attention Mask, which identifies the valid bits and padding bits in the sequence, and is used to mask the padding bits during model calculation.

[0146] Specifically, in this embodiment, requests on the terminal device are typically sparse, but may also arrive concurrently within a short period, such as application startup or batch translation. A small time window is set, for example: When the first request arrives, it is not executed immediately, but a timer is started. Other requests arriving within the same timeframe are merged. Since the different requests have varying lengths, they need to be expanded to the same length, which facilitates their conversion into regular tensors. The merged tensor is then input into the terminal device's GPU, leveraging the GPU's parallel computing capabilities to derive the results of all requests at once. Compared to serial execution, throughput is significantly improved.

[0147] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A lightweight large-scale model running method based on edge deployment, characterized in that, include: S1: Obtain device operation data by real-time monitoring of the terminal device's operating status during the large model's runtime, and store the model file in a mixed precision format according to the quantization precision supported by the terminal device; S2: Determine a mixed-precision quantization strategy by taking the performance and power consumption of the terminal device as constraints, perform strategy analysis on the device operation data and generate an adaptive quantization strategy according to the mixed-precision format, and execute the adaptive quantization strategy. S3: Identify the first user request with continuous semantic relevance in a multi-turn dialogue, and use KV Cache to retain the Key-Value cache file obtained by model inference of the first user request in the historical dialogue; The Key-Value cache file includes a first K vector and a first V vector corresponding to N tokens generated by model inference; S4: When a new second user request is received in the multi-turn dialogue, calculate the semantic embedding vector of the second user request, calculate the cosine similarity between the semantic embedding vector and the first V vector in the Key-Value cache file, and determine whether the second user request has a similar intent based on the calculation result; S5: If the intent is similar, then incremental reasoning is performed on the second user request by reusing the Key-Value cache file. The incremental reasoning process includes: S51: The large model generates output token by token according to the second user request, performs a linear transformation on the current token to obtain the second Q vector, the second K vector, and the second V vector; and reads all the first K vectors and first V vectors in the Key-Value cache file, with all the first K vectors forming the K matrix and all the first V vectors forming the V matrix. S52: Calculate the attention score based on the second Q vector and the K matrix. After normalization, the attention score is weighted and summed in the V matrix to obtain the context vector. S53: Input the context vector into the FFN layer of the large model, calculate and output the logits vector, and convert the logits vector into a probability distribution; S54: Sample the probability distribution to obtain a new token, and append the second K vector and the second V vector to the Key-Value cache file, incrementing the newly generated token sequence by one; Repeat steps S51 to S54 until the newly obtained token is the predefined end marker.

2. The lightweight large model running method based on edge deployment according to claim 1, characterized in that: The quantization precision supported by the terminal device includes at least a first precision and a second precision, wherein the first precision is less than the second precision; the model file setup steps include: S11: Calculate the quantization residual of the large model at the second precision using the first precision as the baseline parameter; S12: Obtain the model baseline weights stored with the first precision and the model residual weights stored with the quantized residuals. The model file stores the model baseline weights and model residual weights to form a model weight matrix.

3. The lightweight large model running method based on edge deployment according to claim 2, characterized in that: The hybrid precision quantization strategy includes a high-load mode, a balanced mode, and a low-power mode. The mode selection for the hybrid precision quantization strategy is performed on the device operating data according to the constraints. The quantization strategies under different modes include: When the high-load mode is selected, the model baseline weights are superimposed with the model residual weights to reconstruct the first model weights, and the activation value of the first model weights is obtained. The model weights are then loaded for model quantization. When the balance mode is selected, the model baseline weights are used as the second model weights and the activation values ​​of the second model weights are obtained. The second model weights are then loaded for model quantization. When the low-power mode is selected, after model preprocessing, the model baseline weights are used as the third model weights and the activation values ​​of the third model weights are obtained. The third model weights are then loaded for model quantization. The model preprocessing is either model pruning or adding an early termination mechanism.

4. The lightweight large model running method based on edge deployment according to claim 3, characterized in that: In the balanced mode, the SIMD instruction set is used to accelerate model quantization, and the steps include: S211: Select the appropriate SIMD instruction set according to the hardware platform of the terminal device. The SIMD instruction set includes at least SIMD loading instructions, SIMD unpacking instructions, SIMD multiplication instructions, and SIMD accumulation instructions. S212: Package multiple second model weights and their activation values, and load the packaged data into the SIMD register at once using the SIMD loading instruction; S213: Decompress the packed data into register data with the second precision data type using the SIMD unpacking instruction; S214: Perform parallel operations on the register data using SIMD multiplication instructions, perform SIMD multiplication on the multiple product results obtained from the parallel operations and their corresponding scaling factors, and then output the accumulated results using the SIMD accumulation instructions.

5. The lightweight large model running method based on edge deployment according to claim 3, characterized in that: The model pruning steps include: S221: Determine the pruning point, and add an auxiliary classification head after the network layer at the corresponding position in the model according to the pruning point; S222: After the data stream flows through the network layer, the probability distribution is directly output through the auxiliary classification head and the process jumps to the last layer for model calculation. The steps for adding the early departure mechanism include: S231: Add an early departure module after each network layer, the early departure module being used to calculate the early departure score of the network layer; S232: Evaluate the output of the network layer based on the early termination score. If early termination is decided, output the result of the current network layer in advance; otherwise, continue to propagate to the next network layer for calculation, and repeat this step until the output result is output.

6. The lightweight large model running method based on edge deployment according to claim 1, characterized in that, The identification step of the first user request in step S3 includes: S31: Store the historical semantic fingerprints of the multi-turn dialogue by maintaining an LRU cache pool; S32: Extract the sentence vector of the first user request, and perform dimensionality reduction and hash encoding on the sentence vector to generate a fixed-length semantic fingerprint; S33: Set a semantic similarity threshold, calculate the Hamming distance between the semantic fingerprint and the historical semantic fingerprint, and if the Hamming distance is less than the similarity threshold, then determine the first user request as a semantically similar request.

7. The lightweight large model running method based on edge deployment according to claim 6, characterized in that, The method for calculating cosine similarity in step S4 includes: S41: The LRU cache pool also stores the Key-Value cache file, and the second user request is converted into the semantic embedding vector by the Embedding model; S42: Calculate the cosine similarity between the semantic embedding vector and the first V vector in the Key-Value cache file using a similarity function; wherein, the similarity function is: in, The semantic embedding vector of the second user request. The first V vector in the Key-Value cache file; S43: Set the similarity threshold to 0.8, when If so, it is determined that the second user request has a similar intent.

8. The lightweight large model running method based on edge deployment according to any one of claims 1-7, characterized in that: User requests are preprocessed using dynamic batch processing. According to steps S1-S5, multiple user requests in the same batch within a short period are concurrently inferred. This dynamic batch processing step includes: S6: Set a time window, merge the user requests received within the time window into a dynamic batch, and perform consistency processing on the length of the sequence in the dynamic batch.

9. The lightweight large-scale model running method based on edge deployment according to claim 8, characterized in that, The length consistency processing steps for the sequences in the dynamic batch described in step S6 include: S61: Set a length threshold and determine whether the length of the sequence in the dynamic batch exceeds the length threshold. If so, truncate the excess part. S62: Find the longest sequence in the dynamic batch, and dynamically fill the relatively short sequences according to the length of the longest sequence; S63: Configure Attention Mask, which identifies the valid bits and padding bits in the sequence, and is used to mask the padding bits during model calculation.

Images