A method and system for large-scale deployment of edge AI all-in-one machines
By establishing shared migration correction information in the edge AI all-in-one machine, the quantization state is dynamically adjusted to adapt to resource changes, solving the storage and calculation problems of historical key-value caching under multiple candidate quantization states, and improving resource utilization efficiency and inference speed.
Patent Information
- Application Number
- CN202611335087.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
AI Technical Summary
In edge AI all-in-one machines, existing technologies struggle to effectively manage the storage overhead and redundant calculations of historical key-value caches under multiple candidate quantization states of large models. This is especially challenging when resource conditions change, as it presents a challenge to adapt historical key-value caches to the target quantization state while controlling additional storage overhead.
By establishing shared migration correction information, joint dimensionality reduction decomposition is performed on the weight differences between different candidate quantization states to generate shared migration correction basis and migration recovery parameters. The quantization state is dynamically adjusted to adapt to the resource changes of the edge AI all-in-one machine, reducing storage overhead and reducing redundant calculations.
This reduces the storage overhead of historical state transition information under multiple candidate quantization states and reduces redundant calculations of regenerating historical key-value caches after quantization state transitions, thereby improving resource utilization efficiency and inference speed.
Smart Images

Figure CN123117345A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of edge artificial intelligence and large model inference technology, and in particular to a method and system for large model quantization deployment of an edge AI all-in-one machine. Background Technology
[0002] With the widespread application of large language models in intelligent question answering, content generation, information retrieval, and industry-specific intelligent applications, deploying large language models on edge devices to reduce network transmission latency, decrease reliance on cloud resources, and improve the localization of data processing has become an important development direction for artificial intelligence applications. However, large language models typically have a large parameter scale, requiring significant computing and storage resources during inference. Edge AI all-in-one machines are limited by hardware conditions such as computing power, memory capacity, and GPU memory capacity. Directly using high-precision models for inference can easily lead to problems such as large model weight space consumption, insufficient available inference cache space, and limited concurrent processing capabilities.
[0003] Model quantization is a common technique for reducing the resource requirements of deploying large models. By using a lower bit width to represent model weights and / or intermediate computational data, the storage space for model weights and the computational overhead during inference can be reduced. In actual operation, the available resources of edge AI all-in-one machines are not always stable. For example, the number of concurrent inference tasks, the length of the input context, and the resource usage of other tasks may all change. Therefore, deployment methods using fixed quantization precision are difficult to continuously adapt to changes in the resources of edge AI all-in-one machines. Existing technologies are gradually adopting methods that adjust the model quantization precision according to the device's operating resource status, enabling the model to perform inference with different quantization states under different resource conditions.
[0004] For large language models that employ autoregressive inference, key-value caches corresponding to historical tokens are typically saved when generating subsequent tokens to avoid repeatedly performing complete calculations on existing contexts. As the number of historical tokens increases, the storage space occupied by the key-value cache continuously increases. During model inference, when adjusting the model weight quantization state related to key-value cache generation, the historical key-value cache already formed before the adjustment corresponds to the quantization weights before the adjustment, while the calculations for subsequent tokens are performed based on the adjusted quantization weights. If existing historical key-value caches are directly used, there may be discrepancies between the historical state and the adjusted model calculation state; if key-value caches are regenerated based on all historical tokens, it will increase redundant calculations and inference latency. Especially when there are multiple candidate quantization states, and the target quantization state needs to be determined during operation based on the real-time resource availability of the edge AI all-in-one machine, there is still room for improvement in how to adapt existing historical key-value caches to the target quantization state while controlling additional storage overhead.
[0005] In the prior art, patent application CN120494006A, entitled "Dynamic Optimization of Large Model Inference Performance and Hardware-Aware Compression Method," discloses a dynamic optimization scheme in the large model inference process. This scheme generates a compression control signal by acquiring the complexity information of the input data and the hardware resource status of the operating platform, and dynamically reconfigures the weights and activation values of the large model according to the compression control signal. The compression control signal may include quantization bit width, sparsity ratio, and operator scheduling strategy, so that the model compression configuration can be adjusted according to the input state and hardware operating state. The above technology demonstrates that dynamically adjusting the quantization or compression state of a large model based on the operating environment has become an existing technical solution in large model deployment. Summary of the Invention
[0006] This invention aims to at least solve one of the aforementioned technical problems existing in the prior art. To this end, this invention provides a method and system for deploying large-scale model quantization in an edge AI all-in-one machine. When a large model has multiple candidate quantization states, it establishes shared migration correction information based on the weight differences between different candidate quantization states. This eliminates the need to save corresponding state migration information for each candidate quantization state when the target quantization state is not yet determined. Furthermore, after the target quantization state is determined during operation, the existing historical key-value cache can be corrected to match the target quantization state. This reduces the storage overhead of historical state migration information under multiple candidate quantization states and reduces redundant calculations caused by regenerating the historical key-value cache after quantization state migration.
[0007] To achieve the above objectives, the present invention adopts the following technical solution.
[0008] A method for large-scale deployment of edge AI all-in-one machines includes:
[0009] Obtain the hardware resource information of the large model to be deployed and the edge AI all-in-one machine, determine the cache coupling quantization unit related to key-value cache generation from the large model to be deployed, and determine the first quantization state and at least two candidate second quantization states of the cache coupling quantization unit based on the hardware resource information.
[0010] The weight difference matrix between the first quantization weight corresponding to the first quantization state and the second quantization weight corresponding to each candidate second quantization state is determined respectively. The multiple weight difference matrices are jointly decomposed to obtain the migration correction basis shared by each candidate second quantization state and the migration recovery parameters corresponding to each candidate second quantization state. The dimension of the migration correction basis is determined according to the weight difference reconstruction error constraint and the storage resource constraint of the quantization migration correction code.
[0011] When the cache-coupled quantization unit is in the first quantization state, historical token reasoning is performed. A historical key-value cache is generated based on the input state entering the cache-coupled quantization unit, and a quantization migration correction code corresponding to the historical key-value cache is generated based on the input state and the migration correction base.
[0012] The system obtains the running resource status of the edge AI all-in-one machine and the current historical token quantity. Based on the weight resource release amount, storage resource occupation amount of the quantization migration correction code and resource occupation amount of the corresponding migration recovery parameters for each candidate second quantization state, the system determines the migration resource benefit parameters corresponding to each candidate second quantization state and determines the target second quantization state from the candidate second quantization states that meet the preset migration benefit conditions.
[0013] The cache-coupled quantization unit is migrated from the first quantization state to the target second quantization state, and the historical state correction amount is determined according to the target migration recovery parameter and quantization migration correction code corresponding to the target second quantization state.
[0014] The historical key-value cache is corrected using the historical state correction amount, and subsequent token inference is performed based on the corrected historical key-value cache and the cache coupled with the quantization unit in the target second quantization state.
[0015] Preferably, the cache-coupled quantization unit includes a key projection unit and / or a value projection unit in the Transformer attention layer; the first quantization state and each candidate second quantization state are characterized by at least one quantization parameter among weighted quantization bit width, quantization scale, and quantization zero point, and at least two candidate second quantization states have different weighted quantization bit widths. Thus, a cache correction relationship across quantization states can be established for the model computation units directly involved in the generation of historical key-value caches, and different candidate second quantization states can correspond to different weight representation accuracies.
[0016] Preferably, the joint dimensionality reduction decomposition of multiple weight difference matrices includes: subtracting the first quantization weight from the second quantization weight corresponding to each candidate second quantization state to obtain the weight difference matrix corresponding to each candidate second quantization state; constructing a joint difference matrix based on the multiple weight difference matrices, and performing low-rank decomposition on the joint difference matrix to obtain the transfer correction basis and the transfer recovery parameters corresponding to each candidate second quantization state; determining the expected storage resource occupancy of the corresponding weight difference reconstruction error and quantization transfer correction code for different candidate dimensions, and determining the candidate dimension that simultaneously satisfies the preset correction error condition and the preset storage resource condition as the dimension of the transfer correction basis. By jointly processing the weight differences corresponding to multiple candidate quantization states, different quantization transfer directions can share the same low-dimensional transfer space without having to establish an independent historical state encoding space for each candidate quantization state.
[0017] Preferably, generating the quantization transition correction code corresponding to the historical key-value cache includes: obtaining the input hidden state of the cache-coupled quantization unit when the historical key-value cache is generated; mapping the input hidden state to a low-dimensional transition space according to the transition correction basis to obtain the quantization transition correction code; establishing the association between the quantization transition correction code and the corresponding historical key-value cache, wherein the association includes at least the model layer identifier, the cache-coupled quantization unit identifier, and the token position identifier; and, if the historical key-value cache is stored in blocks, the association also includes the cache block identifier. Therefore, when the target second quantization state is not yet determined, only one copy of the quantization transition correction code based on the shared transition correction basis can be stored for the same cache-coupled quantization unit and the same historical token, and this quantization transition correction code can be reused when different target second quantization states are subsequently determined.
[0018] Preferably, the migration resource benefit parameter is used to characterize the relationship between the weight resource release capability generated by migrating from the first quantization state to the corresponding candidate second quantization state and the quantization migration auxiliary resource occupation; the quantization migration auxiliary resources include at least the storage resources occupied by the quantization migration correction code corresponding to the current historical token and the storage resources occupied by the corresponding migration recovery parameters; the candidate second quantization state whose migration resource benefit parameter satisfies the preset migration benefit condition is determined as a migratable quantization state. By comprehensively considering the weight resource release capability and the additional resource occupation generated by supporting quantization migration during the quantization state selection process, the matching degree between the quantization migration auxiliary resource investment and the weight resource release capability of the candidate second quantization state can be evaluated based on the changes in the number of historical tokens.
[0019] Preferably, determining the target second quantization state from candidate second quantization states that meet preset migration benefit conditions includes: determining the target resource release requirement based on the operating resource status; screening candidate second quantization states from the migrateable quantization states whose weighted resource release amounts meet the target resource release requirement; and determining the target second quantization state based on the migration resource benefit parameters corresponding to the screened candidate second quantization states, the corresponding weighted differential reconstruction error, and the quantization state migration execution overhead. This allows the selection of the target quantization state to be constrained by both the current resource requirements of the edge AI all-in-one machine and the historical state migration costs.
[0020] Preferably, when the cache-coupled quantization unit is a key projection unit and the large model to be deployed uses a linear positional encoding transformation applied to the key projection result, the key state correction amount before positional encoding is determined according to the quantization transition correction code and the target transition recovery parameter. Then, based on the position identifier of the corresponding historical token, the same linear positional encoding transformation as when generating the corresponding historical key cache is performed on the key state correction amount, and the key state correction amount after the linear positional encoding transformation is superimposed on the corresponding historical key cache. When the cache-coupled quantization unit is a value projection unit, the value state correction amount is determined according to the quantization transition correction code and the target transition recovery parameter, and the value state correction amount is superimposed on the corresponding historical value cache. By processing the key cache and value cache separately, the historical state correction process can be adapted to the generation method of key and value states in attention computation.
[0021] Preferably, correcting the historical key-value cache using historical state correction amounts includes any of the following methods: adding the historical state correction amount to the corresponding historical key-value cache to obtain and store the corrected historical key-value cache; or, retaining the historical key-value cache generated in the first quantization state, and when the historical key-value cache is called in subsequent attention calculations, generating the corresponding historical state correction amount based on the quantization transition correction code and the target transition recovery parameters, adding the historical state correction amount to the called historical key-value cache, and then performing attention calculations. The former method is suitable for scenarios where historical cache updates are completed uniformly after quantization state transitions, while the latter method is suitable for scenarios that reduce the overall rewriting operation of the historical key-value cache.
[0022] The present invention also provides a large-scale model quantization deployment system for an edge AI all-in-one machine, including a quantization state configuration module, a migration parameter generation module, a historical state generation module, a quantization migration decision module, a historical state correction module, and an inference execution module;
[0023] The quantization state configuration module is used to determine the cache-coupled quantization unit related to key-value cache generation from the large model to be deployed, and to determine the first quantization state of the cache-coupled quantization unit and at least two candidate second quantization states.
[0024] The migration parameter generation module is used to determine the weight difference matrix between the first quantization weight corresponding to the first quantization state and the second quantization weight corresponding to each candidate second quantization state, perform joint dimensionality reduction decomposition on multiple weight difference matrices, and determine the dimension of the migration correction basis according to the weight difference reconstruction error constraint and the storage resource constraint of the quantization migration correction code, so as to obtain the migration correction basis shared by each candidate second quantization state and the migration recovery parameters corresponding to each candidate second quantization state respectively.
[0025] The history state generation module is used to generate a history key-value cache based on the input state of the cache-coupled quantization unit when the cache-coupled quantization unit is in the first quantization state, and to generate a quantization migration correction code corresponding to the history key-value cache based on the input state and the migration correction base.
[0026] The quantitative migration decision module is used to determine the migration resource benefit parameters corresponding to each candidate second quantization state based on the running resource status of the edge AI all-in-one machine, the current number of historical tokens, the weight resource release amount corresponding to each candidate second quantization state, the storage resource occupation of the quantization migration correction code, and the resource occupation of the migration recovery parameters, and to determine the target second quantization state from the candidate second quantization states that meet the preset migration benefit conditions.
[0027] The historical state correction module is used to determine the historical state correction amount based on the target migration recovery parameters and quantization migration correction code corresponding to the target second quantization state after the cache-coupled quantization unit migrates to the target second quantization state, and then uses the historical state correction amount to correct the historical key-value cache.
[0028] The inference execution module is used to perform subsequent token inference based on the corrected historical key-value cache and the cache-coupled quantization unit in the target second quantization state.
[0029] Preferably, the large-scale model quantization deployment system further includes a migration state management module; the migration state management module is used to maintain the association between historical key-value cache, quantization migration correction code, cache-coupled quantization unit and current quantization state, and after the corresponding historical key-value cache completes the historical state correction from the first quantization state to the target second quantization state, it releases at least part of the storage resources occupied by the quantization migration correction code of the corresponding historical token, and updates the running resource status of the edge AI all-in-one machine.
[0030] Compared with the prior art, the present invention has at least the following beneficial effects:
[0031] (1) This invention addresses the scenario where the same cache-coupled quantization unit migrates from its current quantization state to multiple candidate quantization states. It performs joint dimensionality reduction processing on the weight differences corresponding to multiple candidate quantization states, enabling multiple candidate quantization states to share a migration correction basis. During the historical token inference stage before the target quantization state is determined, for the same cache-coupled quantization unit and the same historical token, only one quantization migration correction code needs to be generated based on the corresponding shared migration correction basis. There is no need to save the corresponding historical state migration information for each candidate quantization state separately, thereby reducing the additional storage overhead caused by supporting multiple candidate quantization states.
[0032] (2) After determining the target second quantization state during operation, the historical state correction amount can be determined by the migration recovery parameters corresponding to the target second quantization state and the existing quantization migration correction code. The historical key value cache that has been generated in the first quantization state can be adapted and corrected. There is no need to re-execute the complete historical context calculation for all historical tokens, which is conducive to reducing the amount of repeated calculation and inference latency caused by quantization state migration.
[0033] (3) When determining the second quantization state of the target, the weight resource release capability and the occupation of migration auxiliary resources such as quantization migration correction code and migration recovery parameters are included in the migration resource benefit evaluation, so that the quantization migration decision can be adjusted according to the current number of historical tokens and the operating resource status of the edge AI all-in-one machine, thereby improving the effectiveness of quantization state migration under limited storage resources. Attached Figure Description
[0034] Figure 1 This is a flowchart illustrating a method for large-scale model quantization deployment of an edge AI all-in-one machine according to the present invention.
[0035] Figure 2 This is a schematic diagram illustrating the process of generating a shared transition correction base and a quantization transition correction code corresponding to multiple candidate quantization states in this invention.
[0036] Figure 3 This is a flowchart illustrating the process of determining the target quantization state and correcting the historical key-value cache in this invention.
[0037] Figure 4 This is a schematic diagram of the structure of a large-scale model quantization deployment system for an edge AI all-in-one machine according to the present invention. Detailed Implementation
[0038] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make adaptive adjustments to the specific parameters, data accuracy, and implementation form according to the network structure of the large model to be deployed, the hardware configuration of the edge AI all-in-one machine, and the actual quantization method used, without departing from the technical concept of the present invention.
[0039] Example 1
[0040] This embodiment provides a method for deploying large model quantization on an edge AI all-in-one machine. It is applicable to application scenarios where large language models based on the Transformer architecture perform continuous autoregressive inference on an edge AI all-in-one machine, and the model weight quantization state related to key-value cache generation is dynamically adjusted according to the running resource status of the edge AI all-in-one machine.
[0041] like Figure 1As shown, the large-scale model quantization deployment method in this embodiment includes cache-coupled quantization units and candidate quantization state determination, migration parameter generation, historical key-value caching and quantization migration correction code generation, target quantization state determination, historical key-value caching correction, and subsequent token inference.
[0042] First, obtain the hardware resource information of the large model to be deployed and the edge AI all-in-one machine.
[0043] An edge AI all-in-one machine may include a central processing unit (CPU), a graphics processing unit (GPU) and / or a neural network processing unit (NPU), memory, video memory, and model storage devices. Hardware resource information may include available video memory capacity, model weight usage, key-value cache usage, supported data precision, quantization operator support, and the utilization status of corresponding computing resources.
[0044] The large model to be deployed is a Transformer large language model that uses an autoregressive approach to generate tokens. During continuous inference, for historical tokens that have already been computed, the key and value states generated by the attention layer are saved as a key-value cache for use when subsequent tokens undergo attention computation.
[0045] The cache coupling quantization units related to key-value cache generation are identified from the large model to be deployed. In this embodiment, the cache coupling quantization units include key projection units and / or value projection units in the Transformer attention layer.
[0046] With the first Taking the first Transformer layer as an example, the... The hidden input state of a historical token entering this Transformer layer is denoted as:
[0047]
[0048] in, Indicates the Transformer layer number. Indicates the location number of the historical token. The feature dimension represents the input hidden state.
[0049] In the first quantization state, the effective weights corresponding to the key projection units are denoted as:
[0050]
[0051] The effective weights corresponding to the value projection units are denoted as:
[0052]
[0053] in, Indicates the first quantization state. Indicates the key projection output dimension. The dimension of the output value is projected.
[0054] For the cache-coupled quantization unit, at least two candidate second quantization states are pre-configured. For example, the first quantization state can be INT8, and the two candidate second quantization states can be INT6 and INT4, respectively. In other implementations, FP16, BF16, INT8, INT6, INT4, or other data precision states can be used depending on the quantization operator supported by the edge AI all-in-one machine.
[0055] Each quantization state can be characterized by at least one of the following quantization parameters: weighted quantization bit width, quantization scale, and quantization zero point.
[0056] It should be noted that since different quantization states may use different quantization scales and quantization zeros, the first and second quantization weights used to calculate the weight differences are effective weights after being converted to the common computation domain according to the corresponding quantization parameters, rather than directly subtracting the integer encoded values under different quantization states.
[0057] like Figure 2 As shown, after determining the first quantization state and multiple candidate second quantization states, the weight difference matrix corresponding to each candidate second quantization state is determined, and a shared migration correction basis and migration recovery parameters corresponding to each candidate second quantization state are generated based on the multiple weight difference matrices.
[0058] With the first Taking the key projection unit of a Transformer layer as an example, assuming the cache-coupled quantization unit has The candidate second quantization state, the... The effective key projection weights corresponding to the candidate second quantization states are denoted as:
[0059]
[0060] in, Indicates the first One candidate second quantized state.
[0061] The first quantization state is directed to the... The bond projection weight difference matrix corresponding to each candidate second quantization state transition is:
[0062]
[0063] The weight difference matrix corresponding to each candidate second quantization state is obtained in the manner described above.
[0064] Multiple weight difference matrices are combined along the output dimension to form a joint difference matrix:
[0065]
[0066] Joint dimensionality reduction decomposition is performed on the joint difference matrix. This embodiment takes truncated singular value decomposition as an example, yielding:
[0067]
[0068] in, The shared migration correction basis corresponding to the key projection unit of the l-th Transformer layer satisfies:
[0069]
[0070] To share the dimension of the migration correction basis and satisfy ; This is the corresponding migration recovery parameter set.
[0071] Based on the position of the weight difference matrix corresponding to each candidate second quantization state in the joint difference matrix, the migration recovery parameter set is divided into:
[0072]
[0073] in, This represents the migration recovery parameter corresponding to the j-th candidate second quantization state.
[0074] Therefore, each candidate second quantization state satisfies the following:
[0075] The value projection unit is processed in the same way to obtain the shared migration correction basis. and the migration recovery parameters corresponding to each candidate second quantization state. .
[0076] The dimension of the shared migration correction basis is determined jointly based on the weighted differential reconstruction error and the storage resource usage of the quantization migration correction code.
[0077] Specifically, for candidate dimensions We can calculate the weighted differential reconstruction error corresponding to the j-th candidate second quantization state:
[0078]
[0079] in, This represents the weight difference reconstruction error corresponding to the candidate dimension r. This represents the transition correction basis obtained using the candidate dimension r. This indicates the corresponding migration recovery parameters. This represents the Frobenius norm.
[0080] At the same time, based on the configured maximum number of historical tokens N max The candidate dimension *r* and the data storage precision used for the quantization transition correction code are used to estimate the storage resource consumption of the quantization transition correction code. For example, when the quantization transition correction code corresponding to each historical token contains *r* data elements, it can be represented as:
[0081]
[0082] in, This represents the expected storage resource consumption of the quantization transition correction code under candidate dimension r. This represents the number of storage bytes corresponding to each quantization transition correction code element.
[0083] When multiple cache-coupled quantization units and multiple Transformer layers store quantization migration correction codes respectively, the storage resource occupancy of the quantization migration correction codes of each cache-coupled quantization unit in each Transformer layer is determined, and the storage resource occupancy of each storage resource occupancy is summed to obtain the total storage resource occupancy of the current quantization migration correction codes.
[0084] In satisfying:
[0085] And it satisfies:
[0086] The target dimension for the shared migration correction basis is determined from the candidate dimensions.
[0087] in, This represents the upper limit of the allowed weight difference reconstruction error. This indicates the upper limit of storage resources that can be allocated to quantization migration correction codes.
[0088] Therefore, while ensuring that the approximate accuracy of the weighted difference corresponding to each candidate second quantization state meets the requirements, the additional storage resources used to support the subsequent quantization state transition are limited.
[0089] See also Figure 2 When the cache-coupled quantization unit performs historical token reasoning while maintaining the first quantization state, it synchronously generates a historical key-value cache and a quantization migration correction code corresponding to the historical key-value cache.
[0090] For the t-th historical token in the l-th Transformer layer, the key projection unit generates the key projection result before position encoding processing based on the input hidden state and the effective key projection weights in the first quantization state:
[0091]
[0092] When the large model uses rotational position encoding, the corresponding position encoding transformation is performed on the key projection result based on the position parameters corresponding to the t-th historical token:
[0093]
[0094] in, This represents the position encoding transformation corresponding to the historical token position t.
[0095] In this embodiment, the rotational position encoding is a linear position encoding transformation that acts on the key projection result. Therefore, after performing the same position encoding transformation on the key state correction amount before position encoding as on the original key state, it can be superimposed on the historical key cache that has already completed position encoding.
[0096] The value state corresponding to the value projection unit is:
[0097]
[0098] Will It is saved as a historical key-value cache for the corresponding historical token.
[0099] While generating the historical key-value cache, quantization transition correction codes are generated based on the input hidden state and the shared transition correction base.
[0100] The quantization transition correction code corresponding to the key projection unit is:
[0101]
[0102] The quantization transition correction code corresponding to the value projection unit is:
[0103]
[0104] in, and The dimensions of each feature are lower than the feature dimensions of the original input hidden state, so there is no need to continuously save the complete input hidden state, thus retaining the historical state information required for subsequent quantization state transitions.
[0105] A quantization migration correction code is associated with the corresponding historical key-value cache. This association includes at least the Transformer layer number, the cache-coupled quantization unit identifier, and the historical token position number; if the historical key-value cache uses block storage, the association also includes the corresponding key-value cache block number. The cache-coupled quantization unit identifier distinguishes between key projection units, value projection units, and other cache-coupled quantization units within the same Transformer layer, enabling the determination of the quantization migration correction code matching the corresponding historical key-value cache during quantization state transitions.
[0106] Since the quantization transition correction code is obtained based on the transition correction base shared by multiple candidate second quantization states, it is not necessary to predetermine which candidate second quantization state will be used in the future during the historical token inference stage. For the same cache-coupled quantization unit and the same historical token, only one copy of the quantization transition correction code based on the shared transition correction base needs to be stored, without having to store the corresponding quantization transition correction code for different candidate second quantization states.
[0107] like Figure 3 As shown, during the continuous inference process of the large model, the running resource status of the edge AI all-in-one machine and the current number of historical tokens are further obtained, and it is determined whether a quantitative state transition needs to be performed.
[0108] The runtime resource status may include at least one of the following: current available video memory capacity, model weight usage, key-value cache usage, GPU or NPU utilization, and current historical token count.
[0109] Taking available video memory capacity as a resource judgment parameter as an example, the current available video memory capacity is denoted as... The preset safe minimum available video memory is denoted as .
[0110] when: < At that time, determine if there is a storage resource release requirement, and calculate the target resource release requirement:
[0111]
[0112] in, This indicates the minimum amount of storage resources that need to be released to restore the currently available video memory to a preset safe range.
[0113] For the j-th candidate second quantization state, the weight resource release amount is determined based on the weight storage resource occupancy of the first quantization state and the corresponding candidate second quantization state:
[0114]
[0115] in, This indicates the weight storage resource usage of the cache-coupled quantization unit when it uses the first quantization state. This represents the weight storage resource usage of the cache-coupled quantization unit when it adopts the j-th candidate second quantization state.
[0116] Let the current number of historical tokens be... The storage resource usage corresponding to the generated quantization migration correction code is determined based on the current number of historical tokens. And obtain the migration recovery parameter resource usage corresponding to the j-th candidate second quantization state. .
[0117] Based on the weighted resource release amount, the storage resource usage of the quantization migration correction code, and the migration recovery parameter resource usage, the migration resource benefit parameters for the corresponding candidate second quantization state are determined.
[0118] For example, in one specific implementation, the migration resource benefit parameter corresponding to the j-th candidate second quantization state can be expressed as:
[0119]
[0120] in, This represents the migration resource benefit parameter corresponding to the j-th candidate second quantization state.
[0121] The aforementioned migration resource benefit parameter characterizes the relationship between the weight resources that can be released by a candidate second quantization state and the auxiliary resources required to support the migration of that candidate second quantization state. This parameter is only used for resource benefit evaluation during the migration decision-making phase and does not imply that all current quantization migration correction codes must be retained after the quantization state migration is completed.
[0122] Candidate second quantization states that meet the preset migration benefit conditions are determined as migrateable quantization states.
[0123] Furthermore, candidate second quantization states that satisfy the target resource release requirements are selected from the transferable quantization states, i.e., those that satisfy:
[0124]
[0125] When only one candidate second quantization state meets the condition, that candidate second quantization state is determined as the target second quantization state.
[0126] When there are multiple candidate second quantization states that meet the conditions, the selection can be made based on the corresponding migration resource benefit parameters, weight difference reconstruction error, and quantization state migration execution overhead.
[0127] For example, if multiple candidate second quantization states can meet the target resource release requirements, the candidate second quantization state with the smaller weight differential reconstruction error can be selected first; when the weight differential reconstruction errors are the same or similar, the candidate second quantization state with the smaller migration execution overhead or the larger migration resource benefit parameter can be further selected.
[0128] Therefore, the second quantitative state of the target can be dynamically determined during operation based on the actual resource status of the edge AI all-in-one machine, without needing to be fixed in advance before the historical token reasoning begins.
[0129] See also Figure 3After determining the target second quantization state, the cache-coupled quantization unit is migrated from the first quantization state to the target second quantization state, and the historical key-value cache generated in the first quantization state is corrected using the saved quantization migration correction code.
[0130] In this embodiment, when any cache-coupled quantization unit performs a transition from a first quantization state to a target second quantization state, the upstream computation path used to generate the input hidden state of the cache-coupled quantization unit is kept free from quantization state changes that affect the input hidden state. Therefore, the input hidden state of the cache-coupled quantization unit when generating the historical key-value cache in the first quantization state can serve as the computational basis for determining the historical key-value state differences caused by changes in the weight state of the cache-coupled quantization unit itself.
[0131] When it is necessary to adjust upstream computing units that may change the above-mentioned input hidden state at the same time, the quantization state transition can be performed sequentially according to the data processing direction of the large model to be deployed, and the quantization state transition can be performed on the downstream cache-coupled quantization unit after the state processing corresponding to the upstream computing unit is completed; or, the cache-coupled quantization unit whose input hidden state changes due to the change of the upstream quantization state can be excluded from the current quantization transition set.
[0132] Assume the first One candidate second quantization state was determined as the target second quantization state.
[0133] For the key projection unit of the l-th Transformer layer, the effective weights are determined by:
[0134]
[0135] Switch to:
[0136]
[0137] For the t-th historical token already generated in the first quantization state, calculate the key state correction amount before position encoding processing based on the quantization transition correction code and the target transition recovery parameters corresponding to the target second quantization state:
[0138]
[0139] because:
[0140] and:
[0141] therefore:
[0142] When the key cache uses position encoding, the same position encoding transformation as the original historical key cache is performed on the key state correction amount based on the position parameter corresponding to the historical token t:
[0143]
[0144] Subsequently, the key state correction is added to the historical key cache formed in the first quantization state to obtain the corrected key cache:
[0145]
[0146] in, This represents a correction key cache that is adapted to the second quantization state of the target.
[0147] For the value projection unit, the effective weights are changed from: Switch to:
[0148] Determine the value state correction amount based on the quantized migration correction code and the target migration recovery parameters:
[0149]
[0150] Get the correction value cache:
[0151]
[0152] in, This represents a cache of correction values adapted to the second quantization state of the target.
[0153] After the historical key-value cache correction is completed, for newly generated tokens, the key projection weights and value projection weights of the target second quantization state are directly used to generate new key states and value states; for historical tokens that have already been generated, the corrected historical key-value cache is called to perform subsequent attention calculations.
[0154] Therefore, when the target second quantization state needs to be dynamically determined based on the actual operating resources of the edge AI all-in-one machine, for the same cache-coupled quantization unit and the same historical token, only one quantization migration correction code generated based on the shared migration correction base needs to be stored during the historical token inference stage. After determining the specific target second quantization state during operation, the migration recovery parameters corresponding to the target second quantization state are then called to form the corresponding historical state correction amount, without the need to pre-store independent historical state migration information for each candidate second quantization state.
[0155] In one implementation, an explicit correction method can be used. After determining the target second quantization state, the historical key-value cache that needs to continue to be used is uniformly corrected, and the corrected historical key-value cache is written into the storage area corresponding to the original key-value cache or the new key-value cache area.
[0156] After explicit correction is completed, if the corresponding quantization transition correction code is no longer used for other candidate quantization state transitions, the storage resources occupied by the corresponding quantization transition correction code can be released.
[0157] In another implementation, an online correction method can be used, where instead of rewriting all historical key-value caches at once during quantization state transition, the corresponding historical state correction amount is generated based on the quantization transition correction code and the target transition recovery parameters when the corresponding historical key-value cache is actually called during subsequent attention calculation.
[0158] For example, for the first The query status corresponding to each subsequent token When the attention calculation calls the key cache corresponding to the t-th historical token, it can be done as follows: As the valid key state of the corresponding historical token, and will: This serves as the valid value state of the corresponding historical token.
[0159] When using the online correction method, the quantization migration correction code corresponding to the historical key-value cache that has not yet completed the explicit state transition is retained; when the corresponding historical key-value cache is evicted, invalidated, or has completed the explicit correction, the storage resources occupied by the corresponding quantization migration correction code are released.
[0160] After the quantization state transition and historical key-value cache correction are completed, the running resource status of the edge AI all-in-one machine is reacquired. If the current running resource status has been restored to the preset range, the target second quantization state is used to continue performing subsequent inference; if the running resource status still does not meet the preset conditions, the resource release requirements can be re-determined based on the updated running resource status, and subsequent quantization state transitions can be performed on other transferable quantization units.
[0161] In this way, when there are multiple candidate second quantization states and the final target quantization state needs to be determined during operation, it is not necessary to save the corresponding historical state migration data for each candidate second quantization state separately. After the target second quantization state is determined, the existing historical key-value cache can be corrected by using the shared migration correction code and the migration recovery parameters corresponding to the target state, thereby reducing the storage overhead of historical migration information corresponding to multiple candidate quantization states and reducing the redundant calculations caused by re-executing all historical token prefill calculations due to changes in quantization state.
[0162] Example 2
[0163] This embodiment provides a large-scale model quantization deployment system for an edge AI all-in-one machine.
[0164] like Figure 4As shown, the large-scale model quantization deployment system includes a quantization state configuration module, a migration parameter generation module, a historical state generation module, a quantization migration decision module, a historical state correction module, and an inference execution module.
[0165] The quantization state configuration module is used to obtain hardware resource information of the large model to be deployed and the edge AI all-in-one machine, determine the cache coupling quantization unit related to key-value cache generation from the large model to be deployed, and configure the first quantization state and at least two candidate second quantization states for the cache coupling quantization unit.
[0166] The cache-coupled quantization unit may include at least one of the key projection unit and value projection unit in the Transformer attention layer.
[0167] The first quantization state and each candidate second quantization state can be represented by different weighted quantization bit widths.
[0168] The migration parameter generation module is used to obtain the first quantization weight corresponding to the first quantization state and the second quantization weight corresponding to each candidate second quantization state, determine the weight difference matrix between the first quantization weight and each second quantization weight, and perform joint dimensionality reduction decomposition on multiple weight difference matrices to obtain the migration correction basis shared by each candidate second quantization state and the migration recovery parameters corresponding to each candidate second quantization state.
[0169] The migration parameter generation module is also used to determine the corresponding weight differential reconstruction error and the expected storage resource consumption of the quantization migration correction code for different shared migration correction basis candidate dimensions, and to determine the dimension of the shared migration correction basis accordingly.
[0170] The history state generation module is used to perform historical token reasoning when the cache-coupled quantization unit is in the first quantization state. It generates a historical key-value cache based on the input hidden state entering the cache-coupled quantization unit, and generates a quantization migration correction code corresponding to the historical key-value cache using the input hidden state and the shared migration correction base.
[0171] The historical state generation module is also used to establish the association between the quantization migration correction code and the historical key-value cache. The association includes at least the model layer number, the cache-coupled quantization unit identifier, and the historical token position number; when the historical key-value cache uses block storage, the association also includes the corresponding cache block number.
[0172] The quantization migration decision module is used to obtain the running resource status of the edge AI all-in-one machine and the current historical token quantity, and to determine the migration resource benefit parameters corresponding to each candidate second quantization state based on the weight resource release amount corresponding to each candidate second quantization state, the storage resource occupation of the current quantization migration correction code, and the resource occupation of the corresponding migration recovery parameters.
[0173] The quantization migration decision module further determines the target second quantization state from the candidate second quantization states that meet the migration benefit conditions based on the current resource release requirements, and controls the corresponding cache-coupled quantization unit to migrate from the first quantization state to the target second quantization state.
[0174] The historical state correction module is used to determine the historical state correction amount based on the migration recovery parameters corresponding to the target second quantization state and the quantization migration correction code corresponding to the historical token after the cache-coupled quantization unit migrates to the target second quantization state, and to use the historical state correction amount to correct the historical key-value cache formed in the first quantization state.
[0175] When the cache coupling quantization unit is a key projection unit and the large model to be deployed adopts a linear positional encoding transformation implemented on the key projection result, the historical state correction module is also used to perform the same linear positional encoding transformation on the key state correction amount before positional encoding as when generating the corresponding historical key cache, based on the position number of the historical token, and to superimpose the key state correction amount after linear positional encoding transformation onto the corresponding historical key cache.
[0176] The inference execution module is used to continue executing subsequent token inference based on the corrected historical key-value cache and the cache-coupled quantization unit in the target second quantization state.
[0177] Large-scale model quantization deployment systems can also include a migration status management module.
[0178] The migration state management module is used to maintain the relationship between historical key-value cache, quantization migration correction code, model layer number, historical token position, cache block number, and the current quantization state of cache-coupled quantization unit.
[0179] When the corresponding historical key-value cache is explicitly corrected, or when the historical key-value cache is eliminated or expired, the migration status management module releases the storage resources occupied by the corresponding quantization migration correction code and updates the running resource status of the edge AI all-in-one machine so that the quantization migration decision module can perform subsequent quantization status migration judgments.
[0180] The quantization state configuration module, migration parameter generation module, historical state generation module, quantization migration decision module, historical state correction module, inference execution module, and migration state management module can be implemented by the processor in the edge AI all-in-one machine executing the program instructions stored in the memory, or they can be implemented by software programs in collaboration with GPUs, NPUs, or other artificial intelligence acceleration hardware.
[0181] The specific processing procedures executed by each module in this embodiment can be referred to the corresponding method steps in Embodiment 1, and will not be repeated here.
[0182] The above embodiments are merely preferred embodiments of the present invention. Those skilled in the art, based on the disclosure of the present invention, can make corresponding adjustments or substitutions according to specific large model architectures, key-value cache organization methods, quantization methods, and edge AI all-in-one machine hardware structures. As long as these adjustments do not depart from the essence of the technical solution of the present invention, they should all fall within the protection scope of the present invention.
Claims
1. A method for large-scale deployment of edge AI all-in-one machines, characterized in that, include: Obtain the hardware resource information of the large model to be deployed and the edge AI all-in-one machine, determine the cache coupling quantization unit related to key-value cache generation from the large model to be deployed, and determine the first quantization state and at least two candidate second quantization states of the cache coupling quantization unit based on the hardware resource information. The weight difference matrix between the first quantization weight corresponding to the first quantization state and the second quantization weight corresponding to each candidate second quantization state is determined respectively. The multiple weight difference matrices are jointly decomposed to obtain the migration correction basis shared by each candidate second quantization state and the migration recovery parameters corresponding to each candidate second quantization state respectively. The dimension of the migration correction basis is determined according to the weight difference reconstruction error constraint and the storage resource constraint of the quantization migration correction code. When the cache-coupled quantization unit is in the first quantization state, historical token reasoning is performed. A historical key-value cache is generated based on the input state entering the cache-coupled quantization unit, and a quantization migration correction code corresponding to the historical key-value cache is generated based on the input state and the migration correction base. The system obtains the running resource status and current historical token quantity of the edge AI all-in-one machine. Based on the weight resource release amount corresponding to each candidate second quantization state, the storage resource occupation amount of the quantization migration correction code, and the resource occupation amount of the corresponding migration recovery parameter, the system determines the migration resource benefit parameter corresponding to each candidate second quantization state, and determines the target second quantization state from the candidate second quantization states that meet the preset migration benefit conditions. The cache-coupled quantization unit is migrated from the first quantization state to the target second quantization state, and the historical state correction amount is determined according to the target migration recovery parameter corresponding to the target second quantization state and the quantization migration correction code. The historical key-value cache is corrected using the historical state correction amount, and subsequent token inference is performed based on the corrected historical key-value cache and the cache coupled quantization unit in the target second quantization state.
2. The method for large-scale model quantization deployment of the edge AI all-in-one machine according to claim 1, characterized in that, The cache-coupled quantization unit includes at least one of the key projection unit and the value projection unit in the Transformer attention layer; The first quantization state and each candidate second quantization state are characterized by at least one of the following quantization parameters: weighted quantization bit width, quantization scale, and quantization zero point, and at least two candidate second quantization states have different weighted quantization bit widths.
3. The method for large-scale model quantization deployment of the edge AI all-in-one machine according to claim 1, characterized in that, The joint dimensionality reduction decomposition of the multiple weight difference matrices includes: Subtract the first quantization weight from the second quantization weight corresponding to each candidate second quantization state to obtain the weight difference matrix corresponding to each candidate second quantization state. A joint difference matrix is constructed based on the multiple weight difference matrices, and the joint difference matrix is decomposed into a low-rank matrix to obtain the migration correction basis and the migration recovery parameters corresponding to each candidate second quantization state. For different candidate dimensions, the corresponding weighted differential reconstruction error and the expected storage resource usage of the quantization transfer correction code are determined respectively, and the candidate dimension that simultaneously satisfies the preset correction error condition and the preset storage resource condition is determined as the dimension of the transfer correction basis.
4. The method for large-scale model quantization deployment of the edge AI all-in-one machine according to claim 1, characterized in that, The generation of the quantization migration correction code corresponding to the historical key-value cache includes: Obtain the input hidden state of the cache coupling quantization unit when the historical key-value cache is generated; The input hidden state is mapped to a low-dimensional transfer space based on the transfer correction basis to obtain the quantization transfer correction code; Establish an association between the quantization migration correction code and the corresponding historical key-value cache, wherein the association includes at least the model layer identifier, the cache-coupled quantization unit identifier, and the token position identifier. When the historical key-value cache is stored in blocks, the association also includes the cache block identifier.
5. The method for large-scale model quantization deployment of the edge AI all-in-one machine according to claim 1, characterized in that, The migration resource benefit parameter is used to characterize the relationship between the weight resource release capability generated by migrating from the first quantization state to the corresponding candidate second quantization state and the quantization migration auxiliary resource occupation. The quantization migration auxiliary resources include at least the storage resources occupied by the quantization migration correction code corresponding to the current historical token and the storage resources occupied by the corresponding migration recovery parameters; The candidate second quantization state that satisfies the preset migration benefit conditions is determined as a migrateable quantization state.
6. The method for large-scale model quantization deployment of the edge AI all-in-one machine according to claim 5, characterized in that, The step of determining the target second quantization state from candidate second quantization states that meet preset migration benefit conditions includes: Determine the target resource release requirement based on the described operational resource status; From the transferable quantizable states, candidate second quantizable states whose weighted resource release amounts satisfy the target resource release requirements are selected; The target second quantization state is determined based on the migration resource benefit parameters, the corresponding weight difference reconstruction error, and the quantization state migration execution overhead corresponding to the candidate second quantization states obtained through screening.
7. The method for large-scale model quantization deployment of the edge AI all-in-one machine according to claim 2, characterized in that, When the cache coupling quantization unit is a key projection unit and the large model to be deployed adopts a linear positional encoding transformation implemented on the key projection result, the key state correction amount before positional encoding is determined according to the quantization migration correction code and the target migration recovery parameter, and the same linear positional encoding transformation as when generating the corresponding historical key cache is performed on the key state correction amount according to the position identifier of the corresponding historical token, and the key state correction amount after linear positional encoding transformation is superimposed on the corresponding historical key cache. When the cache-coupled quantization unit is a value projection unit, the value state correction amount is determined according to the quantization migration correction code and the target migration recovery parameter, and the value state correction amount is superimposed on the corresponding historical value cache.
8. The method for large-scale model quantization deployment of the edge AI all-in-one machine according to claim 1, characterized in that, The modification of the historical key-value cache using the historical state modification amount includes any of the following methods: The historical state correction amount is added to the corresponding historical key-value cache to obtain and store the corrected historical key-value cache. or, The historical key-value cache generated in the first quantization state is retained. When the historical key-value cache is called in subsequent attention calculations, the corresponding historical state correction amount is generated according to the quantization migration correction code and the target migration recovery parameter. The historical state correction amount is then added to the called historical key-value cache before the attention calculation is performed.
9. A large-scale model quantization deployment system for an edge AI all-in-one machine, characterized in that, include: The quantization state configuration module is used to determine the cache-coupled quantization unit related to key-value cache generation from the large model to be deployed, and to determine the first quantization state of the cache-coupled quantization unit and at least two candidate second quantization states. The migration parameter generation module is used to determine the weight difference matrix between the first quantization weight corresponding to the first quantization state and the second quantization weight corresponding to each candidate second quantization state, perform joint dimensionality reduction decomposition on the multiple weight difference matrices, and determine the dimension of the migration correction basis according to the weight difference reconstruction error constraint and the storage resource constraint of the quantization migration correction code, so as to obtain the migration correction basis shared by each candidate second quantization state and the migration recovery parameters corresponding to each candidate second quantization state respectively. The history state generation module is used to generate a history key-value cache based on the input state entering the cache coupling quantization unit when the cache coupling quantization unit is in the first quantization state, and to generate a quantization migration correction code corresponding to the history key-value cache based on the input state and the migration correction base. The quantization migration decision module is used to determine the migration resource benefit parameters corresponding to each candidate second quantization state based on the running resource status of the edge AI all-in-one machine, the current number of historical tokens, the weight resource release amount corresponding to each candidate second quantization state, the storage resource occupation of the quantization migration correction code, and the resource occupation of the migration recovery parameters, and to determine the target second quantization state from the candidate second quantization states that meet the preset migration benefit conditions. The historical state correction module is used to determine the historical state correction amount based on the target migration recovery parameters and quantization migration correction code corresponding to the target second quantization state after the cache coupling quantization unit migrates to the target second quantization state, and to correct the historical key-value cache using the historical state correction amount. The inference execution module is used to perform subsequent token inference based on the corrected historical key-value cache and the cache-coupled quantization unit in the target second quantization state.
10. The large-scale model quantization deployment system for the edge AI all-in-one machine according to claim 9, characterized in that, It also includes a migration status management module; The migration state management module is used to maintain the association between historical key-value cache, quantization migration correction code, cache-coupled quantization unit and current quantization state. After the corresponding historical key-value cache completes the historical state correction from the first quantization state to the target second quantization state, it releases the storage resources occupied by at least part of the quantization migration correction code of the corresponding historical token and updates the running resource status of the edge AI all-in-one machine.
Citation Information
Patent Citations
Large model reasoning efficiency dynamic optimization and hardware sensing compression method
CN120494006A
