Large model end-side reasoning optimization method and device, equipment and storage medium

Abstract

The invention relates to a large-model end-side reasoning optimization method and device, equipment and a storage medium. The method comprises the steps that a dynamic sparse proportion is predicted according to user input information of a current layer of a large model, a generated standard attention matrix is reconstructed based on the dynamic sparse proportion, a sparse attention matrix is obtained, and the standard attention matrix is generated based on the user input information; performing dynamic step quantization on weight parameters of the sparse attention matrix based on back propagation, performing mixed precision distribution on the weight parameters of different layers according to a quantization result, and performing large model reasoning calculation based on the weight parameters after precision adjustment to generate a key-value cache matrix; and performing real-time compression on the key-value cache matrix based on entropy coding. The end-side reasoning calculation complexity is lower, the adaptive capacity is higher, the reasoning precision is higher, and the occupied memory is less.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to large model edge-side inference optimization methods, apparatus, devices and storage media. Background Technology

[0002] In the field of large-model edge inference, existing technologies have proposed various solutions for attention mechanisms, model quantization, and key-value cache management. Regarding attention mechanism computation, while Longformer reduces some complexity, it struggles to adapt to dynamic sequence lengths. Subsequent attempts have improved this by introducing randomized sparsity, but this relies on predefined sparsity parameters, making it difficult to adaptively adjust based on input data characteristics during inference, further limiting computational efficiency. For model quantization, static quantization cannot adapt to the dynamic input distribution of edge devices; while dynamic quantization addresses the problem of static quantization by calculating scaling factors in real time, the added computational overhead makes it difficult to meet the real-time requirements of edge devices and hinders deployment. In key-value cache management, existing compression methods use low-rank approximation techniques to obtain block-level pruning strategies, but this sacrifices numerical accuracy or fails to flexibly adapt to the feature distribution of the input sequence, resulting in limited memory savings.

[0003] It is evident that existing technologies for edge inference in large models suffer from poor dynamic adaptability, low error control precision, and low memory efficiency. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for optimizing large-model edge inference, in order to solve the technical problems of poor dynamic adaptability, low error control accuracy, and low memory efficiency in the existing technology of large-model edge inference.

[0005] According to one aspect of the embodiments of this application, this application provides a large model edge-side inference optimization method, the method comprising: predicting a dynamic sparsity ratio based on user input information of the current layer of the large model; reconstructing an existing standard attention matrix based on the dynamic sparsity ratio to obtain a sparse attention matrix, wherein the standard attention matrix is ​​generated based on the user input information; performing dynamic step-size quantization on the weight parameters of the sparse attention matrix based on backpropagation; performing mixed precision allocation on the weight parameters of different layers based on the quantization result; performing large model inference calculation based on the precision-adjusted weight parameters to generate a key-value cache matrix; and performing real-time compression on the key-value cache matrix based on entropy coding.

[0006] Optionally, before predicting the dynamic sparsity ratio based on the user input information of the current layer of the large model and reconstructing the generated standard attention matrix based on the dynamic sparsity ratio, the method further includes: obtaining the user input information of the large model, the user input information including user input text; converting the user input text into multiple text minimal semantic units through a word segmenter, performing vector representation on the multiple text minimal semantic units to generate a user semantic input vector sequence, the user semantic input vector sequence including query vectors, key vectors and value vectors; performing a linear transformation on the user semantic input vector sequence to generate the standard attention matrix; calculating the standard attention matrix based on the query vector matrix, the key vector matrix and the vector dimension, wherein the query vector matrix is ​​constructed based on all the query vectors in the user semantic input vector sequence, and the key vectors are constructed based on all the key vectors in the user semantic input vector sequence.

[0007] Optionally, the step of predicting the dynamic sparsity ratio based on the user input information of the current layer of the large model, and reconstructing the generated standard attention matrix based on the dynamic sparsity ratio to obtain a sparse attention matrix, includes: concatenating features of the query vector matrix and the key vector matrix of the current layer of the large model to obtain concatenated features; inputting the concatenated features into the sparsity ratio prediction model for calculation to obtain the dynamic sparsity ratio; and based on the dynamic sparsity ratio, reconstructing the sparse attention matrix for each row of the calculated standard attention matrix. Top-k The maximum value index is calculated to generate a binary sparse mask; the calculated standard attention matrix is ​​then sparsely processed based on the binary sparse mask to obtain the sparse attention matrix.

[0008] Optionally, the step of dynamically quantizing the weight parameters of the sparse attention matrix based on backpropagation, allocating mixed precision to the weight parameters of different layers according to the quantization result, and performing large-scale model inference calculation based on the precision-adjusted weight parameters to generate a key-value cache matrix includes: dynamically adjusting the quantization step size based on a predefined quantization function for the weight parameters, wherein the quantization function is used to perform precision conversion on the weight parameters through the quantization step size; dynamically adjusting the quantization step size based on differentiability constraints, and calculating the gradient of the quantization step size based on the dynamic adjustment result of the quantization step size and the quantization function during backpropagation; updating the quantization step size based on the gradient of the quantization step size according to a chain rule to obtain the quantization result; allocating mixed precision to the weight parameters of different layers according to the quantization result, and performing large-scale model inference calculation based on the value vector matrix, the key vector matrix, and the precision-adjusted weight parameters to generate the key-value cache matrix, wherein the value vector matrix is ​​constructed based on all value vectors.

[0009] Optionally, the step of allocating mixed precision to the weight parameters of different layers based on the quantization result, and performing large-scale model inference calculation based on the value vector matrix, the key vector matrix, and the precision-adjusted weight parameters to generate a key-value cache matrix includes: reducing the precision of the weight parameters based on the quantization result to obtain a first precision and a second precision of the weight parameters, wherein the first precision is higher than the second precision; matching the weight parameters of the first precision to the residual connection layer of the large model during deployment, and matching the weight parameters of the second precision to other layers of the large model; performing large-scale model inference calculation based on the key vector matrix, the value vector matrix, and the precision-adjusted weight parameters to generate the key-value cache matrix, wherein the key-value cache matrix includes the key vector, the value vector, or a combination of the key vector and the value vector of the smallest semantic unit of the text.

[0010] Optionally, the step of real-time compression of the key-value cache matrix based on entropy coding and data decompression based on a pre-constructed scaling function during decompression includes: calculating the probability distribution of each row of the key-value cache matrix; and compressing each row of the key-value cache matrix based on the probability distribution using entropy coding to obtain compressed data of the key-value cache matrix.

[0011] Optionally, after the real-time compression of the key-value cache matrix based on entropy coding, the method further includes: when the large model is decompressed, decoding the compressed data through entropy decoding to obtain an approximate value of the key-value cache matrix; constructing a scaling function based on the approximate value of the key-value cache matrix and a scaling factor, and scaling and reconstructing the original data using the scaling function on the approximate value of the key-value cache matrix, wherein the scaling factor includes the maximum value in the key-value cache matrix.

[0012] According to another aspect of the embodiments of this application, this application provides a large model edge-side inference optimization device, the device comprising: a sparsity processing module, configured to predict a dynamic sparsity ratio based on user input information of the current layer of the large model, and reconstruct an existing standard attention matrix based on the dynamic sparsity ratio to obtain a sparse attention matrix, wherein the standard attention matrix is ​​generated based on the user input information; a weight adjustment module, configured to perform dynamic step-size quantization on the weight parameters of the sparse attention matrix based on backpropagation, perform mixed precision allocation on the weight parameters of different layers according to the quantization result, and perform large model inference calculation based on the precision-adjusted weight parameters to generate a key-value cache matrix; and a data compression module, configured to perform real-time compression on the key-value cache matrix based on entropy coding.

[0013] According to another aspect of the embodiments of this application, this application provides a computer device, including: a processor, a memory, and a network interface. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory through the network interface, and the processor executes the machine-readable instructions to perform the steps of the large model edge inference optimization method as described above.

[0014] According to another aspect of the embodiments of this application, this application provides a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the steps of the large model edge inference optimization method.

[0015] Compared with related technologies, the technical solutions provided in this application have the following advantages: This application provides an optimization method for edge-side inference of large models. In the input processing stage of the large model, a dynamic sparse attention mechanism is introduced after receiving user input information. By predicting the dynamic sparsity ratio based on the user input information of the current layer of the large model, that is, adaptively adjusting the sparsity ratio according to the feature distribution of the input sequence before reconstructing the standard attention matrix, the adaptive capability is stronger, which can reduce the computational complexity of attention to a greater extent, while retaining key interaction information, avoiding redundant calculations, and saving computational resources for subsequent inference. In the computation and inference stage of the large model, gradient-aware quantization is introduced. The quantization step size is dynamically adjusted for the sparsified sparse attention matrix through backpropagation, and combined with mixed precision allocation, which can reduce quantization output error and improve the accuracy of edge-side inference. Compression based on entropy coding can reduce the memory occupation of the edge side, thereby improving memory utilization efficiency. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the hardware environment for the large model edge-side inference optimization method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating an optional large-model edge inference optimization method provided according to an embodiment of this application; Figure 3This is a flowchart of step S202 provided according to an embodiment of this application; Figure 4 This is a flowchart of step S204 provided according to an embodiment of this application; Figure 5 This is a schematic diagram of an optional large-model end-side inference optimization device provided according to an embodiment of this application; Figure 6 This is a schematic diagram of an optional computer device structure provided for an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] To address the problems mentioned in the background art, according to one aspect of the embodiments of this application, an embodiment of a large model edge-side inference optimization method is provided.

[0021] like Figure 1 As shown, the above-mentioned large-scale model edge-side inference optimization method can be applied to, for example... Figure 1 The hardware environment shown is described. The system architecture 100 of the hardware environment includes a terminal device 101 and a server 103. The server 103 is connected to the terminal 101 via a network and can be used to provide services to the terminal or clients installed on the terminal. A database 105 can be set up on the server or independently of the server to provide data storage services to the server 103. The network can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0022] Users can use terminal device 101 to interact with server 103 via a network to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, search applications, instant messaging tools, etc. Terminal device 101 can be various electronic devices with a display screen that support web browsing, including but not limited to smartphones, tablets, laptops, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc. Server 103 can be a server providing various services, such as a backend server supporting the pages displayed on terminal device 101.

[0023] It should be noted that the large model edge-side inference optimization method provided in this application embodiment is generally executed by a server and / or terminal device, and correspondingly, the large model edge-side inference optimization device is generally set in the server and / or terminal device.

[0024] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a large-model edge-side inference optimization method provided in an embodiment of the present invention. Taking the execution of a large-model edge-side inference optimization system jointly composed of a host and a device as an example, the large-model edge-side inference optimization method includes the following steps: S202, predict the dynamic sparsity ratio based on the user input information of the current layer of the large model, and reconstruct the generated standard attention matrix based on the dynamic sparsity ratio to obtain a sparse attention matrix, wherein the standard attention matrix is ​​generated based on the user input information.

[0025] In this embodiment, the aforementioned large model can include different types of models used in different fields, such as large models for Natural Language Processing (NLP), Computer Vision, Speech Processing, and Recommendation Systems. The user input information can be in the form of text, speech, video, or images; in this embodiment, it is described in text form. The sparsity ratio is a key indicator for measuring the sparsity of a matrix or model. In this embodiment, based on an attention mechanism, it describes the proportion of non-zero elements in the attention matrix or weight matrix. The standard attention matrix can refer to the weight matrix calculated in the attention mechanism of the large model, representing the degree of attention the query vector pays to the key vector. In this embodiment, the standard attention matrix is ​​constructed based on user input information.

[0026] More specifically, after a user inputs information into a large model, the dynamic sparsity ratio can be predicted based on the user input information during the input processing stage of the large model. In this embodiment, a lightweight network predicts the dynamic sparsity ratio, with the user input information of the current layer of the large model as input. Specifically, this can be the input features extracted after transforming the user input information, i.e., predicting the dynamic dilution ratio based on the input features corresponding to each layer of the large model. Then, the standard attention matrix is ​​diluted and reconstructed based on the calculated dynamic dilution ratio to obtain a sparse attention matrix. Reconstruction involves selecting the weights of the standard attention matrix, retaining only the key weights to focus on key information. Key weights can be selected using a set threshold, for example, a weight of 0.8 is retained. By providing the above-mentioned dynamic sparse attention mechanism, the sparsity ratio can be adaptively adjusted according to the feature distribution of the input sequence (user input information), reducing the computational complexity of attention from... O ( n 2 Reduced to O ( n log n It has stronger dynamic adaptability; at the same time, it can retain key interaction information, avoid redundant calculations, reduce the amount of computation, improve the efficiency of long text processing, and improve the accuracy of large models in understanding user intent and responding to questions.

[0027] S204, Based on backpropagation, the weight parameters of the sparse attention matrix are dynamically quantized with a step size. According to the quantization result, the weight parameters of different layers are allocated with mixed precision. Based on the precision-adjusted weight parameters, large model inference calculation is performed to generate a key-value cache matrix.

[0028] Backpropagation (BP) is a core algorithm in neural network training, used to efficiently calculate the gradient of the loss function with respect to the model parameters. This allows the parameters to be updated via gradient descent, gradually optimizing the model. Specifically, it is based on the chain rule, calculating the gradient of the parameters at each layer by propagating the error (i.e., the gradient of the loss function) backward from the output layer to the input layer.

[0029] Specifically, in the large model computation stage, a gradient-aware quantization strategy is proposed. Based on backpropagation, dynamic step-size quantization is performed on the sparsified weight parameters to adjust the quantization step size, thereby minimizing quantization error and improving model accuracy. Dynamic step-size quantization means that the step size is no longer fixed but dynamically adjusted according to the data distribution. For example, a smaller step size is used in densely distributed data areas to retain more details, while a larger step size is used in sparsely distributed data areas to reduce redundant information. In the quantization results of dynamic step-size quantization, the quantized values ​​are no longer uniformly distributed but dynamically adjusted according to the density of the data distribution. For example, if floating-point numbers are concentrated in the range [-0.5, 0.5], the step size in this range is 0.05, while the step size in other ranges is 0.2.

[0030] More specifically, by optimizing the quantization step size, the current precision of the weight parameters can be converted to other different precisions. The quantization result includes converting the precision of the weight parameters from FP32 to INT8 or / and FP16. During the deployment phase of the large model, different layers of the large model can then be allocated weight parameters based on the converted precisions. The large model inference calculation process can then continue based on the precision-adjusted weight parameters, generating a key-value cache matrix (KV Cache).

[0031] S206, The key-value cache matrix is ​​compressed in real time based on entropy coding.

[0032] Entropy coding is a lossless data compression method based on the principle of information entropy. It achieves efficient compression by assigning bit strings of different lengths to symbols with different probabilities, including but not limited to Shannon coding, Huffman coding, arithmetic coding, and run-length encoding. In this embodiment, during the caching and output stages of a large model, real-time compression of the key-value cache matrix based on entropy coding can reduce memory usage by more than 80% and latency by 85%, better meeting the needs of real-time interaction.

[0033] In this embodiment of the invention, a dynamic sparse attention mechanism is introduced during the input processing stage of the large model after receiving user input information. By predicting the dynamic sparsity ratio based on the user input information of the current layer of the large model, that is, adaptively adjusting the sparsity ratio according to the feature distribution of the input sequence before reconstructing the standard attention matrix, the adaptive capability is stronger, which can reduce the computational complexity of attention to a greater extent, while retaining key interaction information, avoiding redundant calculations, and saving computational resources for subsequent inference. During the computational inference stage of the large model, gradient-aware quantization is introduced. The quantization step size is dynamically adjusted through backpropagation on the sparsified sparse attention matrix, and combined with mixed precision allocation, which can reduce quantization output error and improve the accuracy of edge inference. Compression based on entropy coding can reduce the memory occupation on the edge, thereby improving memory utilization efficiency.

[0034] In some optional embodiments, prior to step S202 above, the method further includes: S102, Obtain the user input information of the large model, the user input information including user input text; S104, the user input text is converted into multiple text minimal semantic units by a word segmenter, and the multiple text minimal semantic units are vectorized to generate a user semantic input vector sequence, which includes a query vector, a key vector, and a value vector. S106, Perform a linear transformation on the user semantic input vector sequence to generate the standard attention matrix; S108, calculate the standard attention matrix based on the query vector matrix, the key vector matrix, and the vector dimension, wherein the query vector matrix is ​​constructed based on all the query vectors in the user semantic input vector sequence, and the key vectors are constructed based on all the key vectors in the user semantic input vector sequence.

[0035] In this embodiment, taking user input information as text type, when a user inputs a task request to the large model, the user input text included in the task request is extracted. The user input text is then segmented using a tokenizer, converting it into multiple minimal semantic units (tokens). During the conversion process, the tokenizer performs segmentation based on predefined segmentation rules, pre-cleansing the text to remove irrelevant characters such as special symbols and extra spaces. Further text standardization is applied, such as unifying capitalization and processing numbers and punctuation. Then, tokenization is performed again, using pre-trained tokenizers to split the cleaned text into a series of tokens. These pre-trained tokenizers include BERT's Word Piece tokenizer and GPT's BPE tokenizer.

[0036] Furthermore, when representing each token as a vector, the process involves a token segmenter mapping each segmented token to a corresponding and unique ID based on the vocabulary, thus obtaining a complete token ID sequence. This token ID sequence is then input into the embedding layer of the larger model. The embedding layer retrieves the corresponding vector representation from the embedding matrix based on the token ID, generating a user semantic input vector. The embedding matrix is ​​learned during model training, with each token ID corresponding to a fixed-dimensional vector. After processing by the embedding layer, the final result is an embedding vector sequence corresponding to the token ID sequence. For example, the token ID sequence [101, 2023, 3045] is mapped to the embedding vector sequence [v1, v2, v3], where v1, v2, and v3 are 768-dimensional vectors. This embedding vector sequence is the user semantic input vector sequence, including query vectors, key vectors, and value vectors.

[0037] Furthermore, the standard attention matrices generated based on the user semantic input vector sequence also include standard attention matrices for Query, Key, and Value, and these generated standard attention matrices for Query, Key, and Value can be cached. Specifically, the linear transformation described above can include: inputting the user semantic input vector sequence into three independent linear layers for transformation to generate standard attention matrices for Query, Key, and Value. Each linear layer has its own learnable weight matrix and bias vector, and the parameters of the linear transformation formula for each layer are different.

[0038] Furthermore, the standard attention matrix is ​​calculated as shown in equation (1): Aij = d k ( Q i K jT (1) in, Q To query the vector matrix, K The key vector matrix, d k For vector dimensions, i and j These represent the rows and columns of the matrix, respectively.

[0039] In this embodiment, by extracting text from user input information and converting it into tokens based on a word segmenter, a standard attention matrix is ​​generated and calculated, which can provide a data foundation for subsequent matrix sparsity analysis.

[0040] In this embodiment, by accurately extracting text from user input information, key content is effectively captured. Based on a word segmenter, it is efficiently converted into a discrete sequence of user semantic input vectors with semantic representation, which helps to improve the structuring degree and processing efficiency of the data. Based on the sequence of user semantic input vectors, a standard attention matrix is ​​generated and calculated, which can deeply explore the inherent relationship and semantic dependency between tokens, accurately measure the importance weight of different tokens in the context, and thus provide solid and accurate data support for subsequent model inference, semantic understanding and other tasks. This significantly enhances the model's ability to capture and parse complex text semantics and improves the accuracy and reliability of the overall task processing.

[0041] In some alternative embodiments, combined with Figure 3 As shown, step S202 above includes: S2021, Perform feature concatenation on the query vector matrix and the key vector matrix of the current layer of the large model to obtain concatenated features; S2022, The splicing features are input into the sparsity ratio prediction model for calculation to obtain the dynamic sparsity ratio; S2023, based on the dynamic sparsity ratio, calculate each row of the standard attention matrix. Top-k The maximum value index is calculated to generate a binary sparse mask; S2024, Based on the binary sparse mask, the calculated standard attention matrix is ​​subjected to sparse processing to obtain the sparse attention matrix.

[0042] The dynamic sparsity ratio α can be predicted using a lightweight network, whose input is the concatenated features of the query vector matrix and key vector matrix of the current layer of the large model. Q;K The output is a scalar, meaning the output is the dynamic sparsity ratio. α The sparsity ratio prediction model is shown in equation (2) below: α=σ ( Wa [ Q;K ] +ba (2) in, σ For the Sigmoid function, Wa and ba For learnable parameters, Wa ∈R1×2 dk , ba ∈R.

[0043] Furthermore, the dynamic sparsity ratio can be calculated based on... α Calculate the standard attention matrix A ijEach line Top-k Maximum index ( k = αn ),in, Top-k Customizable. Calculates the value for each row. Top-k Maximum value indexing allows for greater focus on key data with higher weight, providing greater attention to them. Further, based on each row... Top-k Generate a binary sparse mask using the maximum index. M ∈{0,1} n×n ,in, M ij =1 indicates that it is reserved. A ij , M ij =0 means not to retain. That is, it's based on all rows. Top-k The maximum value index is used to set the selected position to 1 and the rest to 0, thus generating a sparse mask matrix of {0,1}. M Further based on the generated binary sparse mask pairs A ij After sparsity processing, the final sparse attention matrix is: A sparse= M ⊙ A , A This represents the standard attention matrix, where only non-zero elements are computed.

[0044] Furthermore, traditional attention requires calculating all n 2 One element: O ( n 2 This embodiment uses a dynamic sparsity ratio to dilute the standard attention matrix, requiring only computation. n × k non-zero elements ( k n ),and k = αn .when α When =0.3, k ≈0.3 n The computational complexity is O ( n ×0.3 n )=O(0.3 n 2 ).

[0045] In some alternative embodiments, each row can also be processed using Locality Sensitive Hash (LSH) or block sparsity. Top-k The search for the maximum value index is optimized, further reducing the complexity from... O ( n log n ) down to O (log n The overall complexity is reduced to O ( n log n The optimization of the Top-k maximum value index for each row using LSH leverages the near-neighbor search property of LSH to map high-dimensional data to a low-dimensional hash space, making similar data more likely to fall into the same hash bucket, thus quickly narrowing the search range. Specifically, this can include: standardizing each row of the standard attention matrix (or the entire matrix) to ensure consistent numerical ranges and avoid extreme values ​​interfering with the hash mapping; selecting a suitable LSH variant for maximum value search; constructing a hash table and indexing the data; and searching for the Top-k maximum value index for each row. Experimental results show that the measured sequence length... n When the value is 2048, the computational complexity increases from 2048. 2 =4,194,304 operations reduced to 2,048× 0.3×2048 Approximately 1,269,760 operations were performed, a reduction of 69.7%, resulting in a 3.2x increase in inference speed. Combined with LSH optimization, the complexity was further reduced to... O ( n log n The speed is increased by more than 8 times.

[0046] In this embodiment, the dynamic sparsity ratio calculated based on the aforementioned sparse ratio prediction model can be dynamically adjusted according to the feature distribution of the input sequence to adapt to inputs of different lengths and semantic complexities. By preserving key weights through sparsity processing, the model's understanding of user intent and accuracy in responding can be improved; furthermore, it can reduce computational complexity, lowering the computational complexity of attention from... O ( n 2 ) down to O ( n log n This improves the efficiency of processing long texts and avoids redundant calculations caused by fixed sparse patterns.

[0047] In some alternative embodiments, combined with Figure 4 As shown, step S204 above includes: S2041, a predefined quantization function is defined based on the quantization step size and the weight parameters, wherein the quantization function is used to perform precision conversion on the weight parameters through the quantization step size; S2042, The quantization step size is dynamically adjusted based on the differentiability constraint, and the gradient of the quantization step size is calculated during backpropagation based on the dynamic adjustment result of the quantization step size and the quantization function. S2043, Based on the chain rule, the quantization step size is updated according to the gradient of the quantization step size to obtain the quantization result; S2044, Based on the quantization result, the weight parameters of different layers are allocated with mixed precision, and large model inference calculation is performed based on the value vector matrix, the key vector matrix and the precision-adjusted weight parameters to generate the key-value cache matrix, wherein the value vector matrix is ​​constructed based on all value vectors.

[0048] The quantization function, constructed based on the quantization step size and the weight parameters of the sparse attention matrix, is shown in equation (3) below: Q ( w =round( w / s )· s (3) in, s The quantization step size is used to convert the weight parameter w from FP32 to INT8 or FP16 representation.

[0049] Furthermore, during the training phase of a large model, the quantization step size is adjusted using differentiability constraints. s Dynamic adjustments are made, as shown in equation (4) below: s = n · δ ∥ w ∥1(4) in, δ These are hyperparameters used to control the quantization granularity. n It is the number of weight parameters, || w ||1 represents the L1 norm of the weight parameters.

[0050] Furthermore, during backpropagation, the quantization step size is calculated. s The gradient is calculated, and the quantization step size is adjusted using the chain rule. s An update is performed to minimize the quantization error, where the result after updating the quantization step size is used as the quantization result. Specifically, the quantization step size is calculated. s As shown in equation (5): s L= (5) in, L The loss function; i For weighted indexes, i =1,2,…,n This represents the total weight.

[0051] In some examples, the static quantization error is: Δ 静态 =max∣ w Q 静态 ( w The value of Δ is typically determined by the distribution of calibration data. For dynamic quantization error, the dynamically updated Δ is minimized via gradient descent. s Make quantized value Q 动态 ( w Closer to the true value w Experiments on ResNet-50 show that dynamic quantization reduces the output error from 1.2% on static INT8 to 0.8%, a reduction of 33.3%, bringing it closer to the target value (below 30%). On LLaMA-7B, the error is further reduced to 0.6%, improving model accuracy by more than 40%.

[0052] Furthermore, different layers of the large model have varying sensitivities to the precision of the weight parameters based on their computational complexity. Therefore, in this embodiment, a mixed-precision strategy is used to allocate weight parameters of different precisions to different layers of the large model. Then, a key-value cache matrix (KV Cache) is generated by combining the value vector matrix and the key vector matrix to continue executing the large model inference task.

[0053] In this embodiment, a gradient-aware quantization strategy is proposed. By dynamically adjusting the quantization step size through backpropagation, the low-precision quantization error adaptively converges during the training process, thereby reducing information loss and reducing the quantization error to less than 30% of the static quantization error. Combined with mixed-precision hierarchical allocation, the numerical stability of key layers is ensured while accelerating computation. This significantly improves the training speed of large models in resource-constrained environments, and the model convergence and generalization ability are close to the level of full-precision training. Ultimately, this achieves dual optimization of computational efficiency and model accuracy, improving the accuracy and efficiency of edge inference.

[0054] In some optional embodiments, step S2044 specifically includes: Based on the quantization result, the weight parameter is reduced in precision to obtain a first precision and a second precision of the weight parameter, wherein the first precision is higher than the second precision. The first precision weight parameters are matched to the residual connection layer of the large model during deployment, and the second precision weight parameters are matched to the other layers of the large model. Based on the key vector matrix, the value vector matrix, and the precision-adjusted weight parameters, large model inference calculations are performed to generate the key-value cache matrix. The key-value cache matrix includes the key vector, the value vector, or a combination of the key vector and the value vector of the smallest semantic unit of the text.

[0055] In this embodiment, the quantization step size is used to adjust the precision of the weight parameters. Therefore, based on the quantization result updated with the quantization step size, the precision of the weight parameters can be reduced to obtain a first precision and a second precision. Typically, the default precision for traditional training of large models is PF32. In this embodiment, after precision reduction, PF32 can be converted to PF16 and INT8. Of course, it can also be converted to other precisions, such as BF16, INT4, etc., and is not a limiting factor here. PF16 can be used as the first precision, and INT8 as the second precision.

[0056] Furthermore, based on the different sensitivities of different layers to precision, the weight parameters in the residual connection layers during large model deployment can use first precision, while the weight parameters in other layers during large model deployment can use second precision. During deployment, using FP16 precision for the weights in the residual connections preserves a high dynamic range to reduce the risk of gradient vanishing; using INT8 precision for other layers further reduces memory usage. Through a hierarchical quantization strategy, combined with the different sensitivities of different layers to quantization errors, the cumulative effect of errors is significantly reduced.

[0057] Furthermore, the generated key-value cache matrix C∈R n×dk each line C i This represents a token's key vector, value vector, or a combination of both, where the combination can be based on predefined rules. n The sequence length is given.

[0058] In this embodiment, when deploying a large model, using high-precision first-precision weights for the residual connection layers preserves the high dynamic range of the residual connection layers, thereby retaining key gradient information within these layers. Simultaneously, using lower-precision second-precision weights for other layers significantly reduces computational overhead and memory usage. This layered precision allocation strategy achieves an optimal balance between computational efficiency and numerical accuracy with almost no loss to the overall model performance.

[0059] In some optional embodiments, step S206 above includes: S2061, Calculate the probability distribution of each row of the key-value cache matrix; S2062, Based on the probability distribution, each row of the key-value cache matrix is ​​compressed using the entropy encoding to obtain compressed data of the key-value cache matrix.

[0060] In this embodiment, the entropy encoding described above is Huffman encoding. First, the KV Cache matrix C∈R is calculated. n×dk each line C i The probability distribution is shown in equation (6) below: p ( c i )=∑ j exp( c j )exp( c i (6) in, c i For the first i The vector representation of rows utilizes the Softmax function to map numerical values ​​to a probability space, facilitating subsequent entropy coding.

[0061] Furthermore, based on probability distribution p ( c i ), using Huffman coding C i Compress it. C comp = Huffman( p ( c i Finally, the compressed data C is obtained. comp ∈Rm, where m n, the length of the compressed data m ≈ ∑ p ( c i log2 p ( c i (bits). In this embodiment, Huffman coding further improves compression efficiency by assigning short codewords to high-frequency values ​​and long codewords to low-frequency values.

[0062] In some examples, through compression ratio derivation and comparison, the original KV cache matrix size is... n × d k ×4 bytes (FP32 storage). Compressed size: m × n Bits (Huffman coding), where, m ≈ ∑p ( c i log2 p ( c i Lower bound of entropy: If p ( c i The distribution is uniform, and the entropy H( p )=log2 d k Compression ratio nd k ×4 m ≈32log2 d k .For example, d k At a compression ratio of 64, the theoretical compression ratio is 326 = 0.1875 (i.e., 18.75% of the original size). However, in practice... p ( c i Non-uniform, compression ratio can reach 10%-20%. Based on actual measurements, the measured sequence length... n =1024, d k When the value is 64, the original KV Cache matrix size is 1024×64×4=262144 bytes (256KB), and after compression, it only requires 200 bits (about 25 bytes), reducing memory usage by 99.06%.

[0063] In this embodiment, during the caching and output stages of the large model, the key-value cache matrix generated during the inference process is compressed in real time based on entropy coding. After probability distribution modeling and Huffman coding, memory usage can be reduced, even by more than 80%.

[0064] In some optional embodiments, after step S206 above, the method further includes: When the large model is decompressed, the compressed data is decoded by entropy decoding to obtain an approximate value of the key-value cache matrix; A scaling function is constructed based on the approximation of the key-value cache matrix and a scaling factor. The approximation of the key-value cache matrix is ​​then reconstructed using the scaling function to scale the original data. The scaling factor includes the maximum value in the key-value cache matrix.

[0065] In this embodiment of the application, entropy decoding is the inverse process of entropy encoding. Essentially, it is a reverse mapping, that is, parsing the compressed bit stream bit by bit according to the rules agreed upon during encoding to restore the symbol (or value) of the original data. Its core function is to restore the compact bit stream compressed by entropy encoding to the original data.

[0066] In this embodiment, scaling and reconstruction during large model decompression includes: during the decompression stage, scaling and reconstruction of Ccomp... i Approximate values ​​are obtained by performing Huffman decoding. c~i The original numerical range was then reconstructed by scaling: C i = c~i ·max( C ), where max( C The maximum value in the original KV cache is calculated by traversing the entire KV cache matrix before compression, and stored as a scalar. This scalar value is used for scaling and reconstruction during decompression. This method preserves the numerical range information, ensuring accuracy in reconstruction error correction. <1%, to meet the accuracy requirements of edge-side inference.

[0067] The decompression process generates decompression errors, which stem from the discretization of Huffman coding. =max∣ C i C ~i | / max( C The decompression error is reconstructed using a scaling factor max( C The numerical range was restored, and the reconstruction error was experimentally measured. =0.8%, <0.01 (1%), while reducing memory usage, it still meets the accuracy requirements of edge inference.

[0068] In this embodiment, by introducing a reconstruction factor to reconstruct the decoding errors generated during decoding, it is possible to ensure that the decoded data recovers to its original numerical range, thus ensuring the accuracy of the cached data and meeting the accuracy requirements of edge inference. Ultimately, this achieves edge inference output with fast computation speed, high data accuracy, and lower memory consumption.

[0069] According to another aspect of the embodiments of this application, such as Figure 5 As shown, corresponding to the large model edge-side inference optimization method in the above embodiments, this embodiment provides a large model edge-side inference optimization device, the device comprising: The sparsity processing module 501 is used to predict the dynamic sparsity ratio based on the user input information of the current layer of the large model, and to reconstruct the generated standard attention matrix based on the dynamic sparsity ratio to obtain a sparse attention matrix, wherein the standard attention matrix is ​​generated based on the user input information. The weight adjustment module 503 is used to perform dynamic step-size quantization on the weight parameters of the sparse attention matrix based on backpropagation, perform mixed precision allocation on the weight parameters of different layers according to the quantization result, and perform large model inference calculation based on the precision-adjusted weight parameters to generate a key-value cache matrix. The data compression module 505 is used to compress the key-value cache matrix in real time based on entropy coding.

[0070] It should be noted that in this embodiment, the sparse processing module 501 can be used to execute step S202 in this application embodiment, the weight adjustment module 503 in this embodiment can be used to execute step S204 in this application embodiment, and the data compression module 505 in this embodiment can be used to execute step S206 in this application embodiment.

[0071] In some optional embodiments, the apparatus further includes: an information acquisition module for acquiring the user input information of a large model, the user input information including user input text; a conversion submodule for converting the user input text into multiple text minimal semantic units through a word segmenter, performing vector representation on the multiple text minimal semantic units, and generating a user semantic input vector sequence, the user semantic input vector sequence including query vectors, key vectors, and value vectors; a matrix generation submodule for performing a linear transformation on the user semantic input vector sequence to generate the standard attention matrix; and a matrix calculation submodule for calculating the standard attention matrix based on the query vector matrix, the key vector matrix, and the vector dimensions, wherein the query vector matrix is ​​constructed based on all the query vectors in the user semantic input vector sequence, and the key vectors are constructed based on all the key vectors in the user semantic input vector sequence.

[0072] In some optional embodiments, the sparse processing module 501 includes: a feature concatenation submodule, used to concatenate the query vector matrix and the key vector matrix of the current layer of the large model to obtain concatenated features; a sparse ratio calculation submodule, used to input the concatenated features into a sparse ratio prediction model for calculation to obtain the dynamic sparse ratio; and a mask generation submodule, used to generate a mask based on the dynamic sparse ratio for each row of the calculated standard attention matrix. Top-k The maximum value index is calculated to generate a binary sparse mask; the sparse processing submodule is used to perform sparse processing on the calculated standard attention matrix based on the binary sparse mask to obtain the sparse attention matrix.

[0073] In some optional embodiments, the weight adjustment module 503 includes: a function definition submodule, used to predefine a quantization function based on a quantization step size and the weight parameters, the quantization function being used to perform precision conversion on the weight parameters through the quantization step size; a step size adjustment submodule, used to dynamically adjust the quantization step size based on differentiability constraints, and to calculate the gradient of the quantization step size during backpropagation based on the dynamic adjustment result of the quantization step size and the quantization function; an update submodule, used to update the quantization step size based on a chain rule and the gradient of the quantization step size to obtain the quantization result; and a precision allocation submodule, used to perform mixed precision allocation on the weight parameters of different layers based on the quantization result, and to perform large model inference calculation based on the value vector matrix, the key vector matrix, and the precision-adjusted weight parameters to generate the key-value cache matrix, wherein the value vector matrix is ​​constructed based on all value vectors.

[0074] In some optional embodiments, the precision allocation submodule is specifically used for: reducing the precision of the weight parameters according to the quantization result to obtain a first precision and a second precision of the weight parameters, wherein the first precision is higher than the second precision; matching the weight parameters of the first precision to the residual connection layer of the large model during deployment, and matching the weight parameters of the second precision to other layers of the large model; performing large model inference calculation based on the key vector matrix, the value vector matrix and the precision-adjusted weight parameters to generate the key-value cache matrix, wherein the key-value cache matrix includes the key vector, the value vector or a combination of the key vector and the value vector of the smallest semantic unit of the text.

[0075] In some optional embodiments, the data compression module 505 includes: a probability calculation submodule for calculating the probability distribution of each row of the key-value cache matrix; and a compression submodule for compressing each row of the key-value cache matrix based on the probability distribution using entropy encoding to obtain compressed data of the key-value cache matrix.

[0076] In some optional embodiments, the apparatus further includes: a decoding module, configured to decode the compressed data by entropy decoding when the large model is decompressed, to obtain an approximate value of the key-value cache matrix; and a scaling module, configured to construct a scaling function based on the approximate value of the key-value cache matrix and a scaling factor, and to reconstruct the original data by scaling the approximate value of the key-value cache matrix using the scaling function, wherein the scaling factor includes the maximum value in the key-value cache matrix.

[0077] It should be noted that the examples and application scenarios implemented by the above modules, sub-modules, and sub-units and their corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, sub-modules, and sub-units, as part of the system, can run in the hardware environment of the large model edge-side inference optimization device, and can be implemented in software or hardware.

[0078] According to another aspect of the embodiments of this application, this application provides a computer device, such as... Figure 6 As shown, it includes a memory 601, a processor 603, a communication interface 605, and a communication bus 606. The memory 601 stores a computer program that can run on the processor 603. The memory 601 and the processor 603 communicate through the communication interface 605 and the communication bus 606. When the processor 603 executes the computer program, it implements the steps of the above-mentioned large model edge inference optimization method.

[0079] The memory and processor in the aforementioned computer equipment communicate with each other via a communication bus and a communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0080] The aforementioned memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0081] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0082] According to another aspect of the embodiments of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the large model edge-side inference optimization method in any of the above embodiments.

[0083] Optionally, in this embodiment, the computer-readable medium is configured to store program code for the processor to execute the steps of the large model edge-side inference optimization method described in the above embodiments, wherein the steps of the large model edge-side inference optimization method specifically include: S202. Predict the dynamic sparsity ratio based on the user input information of the current layer of the large model, and reconstruct the generated standard attention matrix based on the dynamic sparsity ratio to obtain a sparse attention matrix, wherein the standard attention matrix is ​​generated based on the user input information. S204. Based on backpropagation, the weight parameters of the sparse attention matrix are dynamically quantized with a step size. According to the quantization result, the weight parameters of different layers are allocated with mixed precision. Based on the precision-adjusted weight parameters, large model inference calculation is performed to generate a key-value cache matrix. S206. Perform real-time compression on the key-value cache matrix based on entropy coding.

[0084] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here. Furthermore, in the specific implementation of this application embodiment, the above embodiments can be consulted, and corresponding technical effects can be achieved.

[0085] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof. For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. Software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0086] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0087] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the division of modules is merely a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0088] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0089] It should be noted that, in this document, relational terms such as "first," "second," etc., are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprise," "include," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprises a…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0090] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A large-scale model edge-side inference optimization method, characterized in that, The method includes: Based on the user input information of the current layer of the large model, predict the dynamic sparsity ratio, and reconstruct the generated standard attention matrix based on the dynamic sparsity ratio to obtain the sparse attention matrix, wherein the standard attention matrix is ​​generated based on the user input information; The weight parameters of the sparse attention matrix are dynamically quantized with a step size based on backpropagation. The weight parameters of different layers are then allocated with mixed precision based on the quantization results. Finally, large model inference calculations are performed based on the precision-adjusted weight parameters to generate a key-value cache matrix. The key-value cache matrix is ​​compressed in real time based on entropy coding.

2. The large-model end-side inference optimization method according to claim 1, characterized in that, Before predicting the dynamic sparsity ratio based on the user input information of the current layer of the large model, and reconstructing the generated standard attention matrix based on the dynamic sparsity ratio, the method further includes: Obtain the user input information of the large model, the user input information including user input text; The user input text is converted into multiple minimal semantic units by a word segmenter, and the multiple minimal semantic units are represented by vectors to generate a user semantic input vector sequence, which includes a query vector, a key vector, and a value vector. The user semantic input vector sequence is linearly transformed to generate the standard attention matrix; The standard attention matrix is ​​calculated based on the query vector matrix, the key vector matrix, and the vector dimension, wherein the query vector matrix is ​​constructed based on all the query vectors in the user semantic input vector sequence, and the key vectors are constructed based on all the key vectors in the user semantic input vector sequence.

3. The large-model end-side inference optimization method according to claim 2, characterized in that, The step involves predicting a dynamic sparsity ratio based on user input information from the current layer of the large model, and reconstructing the generated standard attention matrix based on the dynamic sparsity ratio to obtain a sparse attention matrix, including: The query vector matrix and the key vector matrix of the current layer of the large model are concatenated to obtain concatenated features; The splicing features are input into the sparsity ratio prediction model for calculation to obtain the dynamic sparsity ratio. Based on the dynamic sparsity ratio, the values ​​of each row in the calculated standard attention matrix are... Top-k The maximum value index is calculated to generate a binary sparse mask; The calculated standard attention matrix is ​​sparsely processed based on the binary sparse mask to obtain the sparse attention matrix.

4. The large-model end-side inference optimization method according to claim 2, characterized in that, The process involves dynamically quantizing the weight parameters of the sparse attention matrix using backpropagation, allocating mixed precision to the weight parameters of different layers based on the quantization results, and performing large-scale model inference calculations based on the precision-adjusted weight parameters to generate a key-value cache matrix. Based on the quantization step size and the predefined quantization function of the weight parameters, the quantization function is used to perform precision conversion on the weight parameters through the quantization step size; The quantization step size is dynamically adjusted based on the differentiability constraint, and the gradient of the quantization step size is calculated during backpropagation based on the dynamic adjustment result of the quantization step size and the quantization function. Based on the chain rule, the quantization step size is updated according to the gradient of the quantization step size to obtain the quantization result; Based on the quantization results, the weight parameters of different layers are allocated with mixed precision, and large model inference calculations are performed based on the value vector matrix, the key vector matrix, and the precision-adjusted weight parameters to generate the key-value cache matrix, wherein the value vector matrix is ​​constructed based on all value vectors.

5. The large-model end-side inference optimization method according to claim 4, characterized in that, The step of performing mixed precision allocation on the weight parameters of different layers based on the quantization result, and generating a key-value cache matrix by performing large model inference calculation based on the value vector matrix, the key vector matrix, and the precision-adjusted weight parameters, includes: Based on the quantization result, the weight parameter is reduced in precision to obtain a first precision and a second precision of the weight parameter, wherein the first precision is higher than the second precision. The first precision weight parameters are matched to the residual connection layer of the large model during deployment, and the second precision weight parameters are matched to the other layers of the large model. Based on the key vector matrix, the value vector matrix, and the precision-adjusted weight parameters, large model inference calculations are performed to generate the key-value cache matrix. The key-value cache matrix includes the key vector, the value vector, or a combination of the key vector and the value vector of the smallest semantic unit of the text.

6. The large-model end-side inference optimization method according to claim 1, characterized in that, The real-time compression of the key-value cache matrix based on entropy coding, and the data decompression based on the constructed scaling function during decompression, include: Calculate the probability distribution of each row of the key-value cache matrix; Based on the probability distribution, each row of the key-value cache matrix is ​​compressed using the entropy encoding to obtain compressed data of the key-value cache matrix.

7. The large-model end-side inference optimization method according to claim 6, characterized in that, After performing real-time compression of the key-value cache matrix based on entropy coding, the method further includes: When the large model is decompressed, the compressed data is decoded by entropy decoding to obtain an approximate value of the key-value cache matrix; A scaling function is constructed based on the approximation of the key-value cache matrix and a scaling factor. The approximation of the key-value cache matrix is ​​then reconstructed using the scaling function to scale the original data. The scaling factor includes the maximum value in the key-value cache matrix.

8. A large-scale model end-side inference optimization device, characterized in that, The device includes: The sparsity processing module is used to predict the dynamic sparsity ratio based on the user input information of the current layer of the large model, and to reconstruct the generated standard attention matrix based on the dynamic sparsity ratio to obtain a sparse attention matrix, wherein the standard attention matrix is ​​generated based on the user input information. The weight adjustment module is used to perform dynamic step-size quantization on the weight parameters of the sparse attention matrix based on backpropagation, allocate mixed precision to the weight parameters of different layers according to the quantization results, and perform large model inference calculation based on the precision-adjusted weight parameters to generate a key-value cache matrix. The data compression module is used to compress the key-value cache matrix in real time based on entropy coding.

9. A computer device, comprising: A processor, a memory, and a network interface, wherein the memory stores machine-readable instructions executable by the processor, characterized in that: when the computer device is running, the processor communicates with the memory via the network interface, and the processor executes the machine-readable instructions to perform the steps of the large model edge inference optimization method as described in any one of claims 1 to 7.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the steps of the large model edge inference optimization method according to any one of claims 1 to 7.

Images