Task-aware and hardware constraint based large language model deployment method and system

Abstract

This invention relates to the field of large language models, providing a method and system for deploying large language models based on task awareness and hardware constraints. The method includes: generating probe data streams from natural language text to obtain a unified analysis context; constructing a second-order covariance matrix based on the unified analysis context, calculating a noise tolerance constant, calculating a basic leverage score based on the noise tolerance constant, and obtaining a global channel sensitivity score set; dividing data into physical data blocks, aggregating channel sensitivity of the physical data blocks to obtain block-level sensitivity; calculating peak memory usage based on hardware constraints, determining a block-level candidate region threshold, dividing block precision based on block-level sensitivity, peak memory usage, and the block-level candidate region threshold, and obtaining block-level precision allocation results; using a global precision mask table and a shared micro-scaling factor to obtain a static deployment descriptor, and deploying the target model based on the static deployment descriptor to complete the large language model deployment.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a method and system for deploying large language models based on task awareness and hardware constraints. Background Technology

[0002] With the widespread application of large language models in natural language processing scenarios such as autonomous driving vehicle-machine interaction, local code generation, and offline medical question answering, how to deploy large language models used to process natural language text data to edge devices with limited physical memory and computing power has become an important technical problem in this field.

[0003] However, the inference process of large language models deployed on edge devices is generally constrained by the "memory wall" problem. On the one hand, the autoregressive generation mechanism causes the key-value cache to grow dynamically with the sequence length, significantly increasing the memory usage; on the other hand, edge devices cannot expand hardware resources due to power consumption and heat dissipation limitations, making memory capacity and bandwidth the core bottlenecks restricting inference performance.

[0004] To alleviate the aforementioned problems, post-training quantization techniques have been widely adopted. Among them, mixed-precision quantization methods, by allocating differentiated bit widths to different channels within the same tensor, balance model accuracy and storage compression to some extent. Meanwhile, next-generation hardware architectures supporting micro-scaling formats provide underlying execution support for low-bit mixed-precision quantization by sharing scaling factors at the data block level.

[0005] However, most existing hybrid precision deployment methods use channels as the basic optimization unit. Their precision allocation strategies, memory layout methods, and execution scheduling mechanisms are independent of each other, lacking a unified structural constraint and collaborative optimization framework, which leads to a disconnect between algorithm layer decisions and hardware execution mechanisms. Summary of the Invention

[0006] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a method and system for deploying large language models based on task awareness and hardware constraints, which enables the deployment of large language models under limited video memory conditions, reduces the latency of large language models, and achieves high-throughput deployment.

[0007] This invention provides a method for deploying large language models based on task awareness and hardware constraints, including: S1: Obtain natural language text, generate probe data stream based on natural language text, use probe data stream to obtain initial physical monitoring view and two-dimensional dedicated activation space topology matrix, and obtain unified analysis context through initial physical monitoring view and two-dimensional dedicated activation space topology matrix; S2: Construct a second-order covariance matrix based on the unified analysis context, calculate the noise tolerance constant of the second-order covariance matrix, calculate the basic leverage score based on the noise tolerance constant, and obtain the global channel sensitivity score set through the basic leverage score; S3: Divide the data into physical data blocks based on the global channel sensitivity score set, and aggregate the channel sensitivity of the physical data blocks to obtain the block-level sensitivity; S4: Obtain hardware constraints, calculate peak video memory usage based on hardware constraints, determine block-level candidate region thresholds, and perform block precision partitioning based on block-level sensitivity, peak video memory usage, and block-level candidate region thresholds to obtain block precision allocation results; S5: Generate a global precision mask table based on the block-level precision allocation results, generate a shared micro-scaling factor, use the global precision mask table and the shared micro-scaling factor to obtain a static deployment descriptor, deploy the target model based on the static deployment descriptor, and complete the deployment of the large language model.

[0008] According to the large language model deployment method based on task awareness and hardware constraints provided by the present invention, in step S1, the natural language text is subjected to word segmentation and encoding processing to obtain an input feature tensor, and the input feature tensor is loaded to obtain the probe data stream.

[0009] According to the large language model deployment method based on task awareness and hardware constraints provided by the present invention, in step S1, the internal activation response tensor is obtained through the probe data stream, a preset continuous step size is determined, the direct memory access controller is scheduled to perform merged memory access according to the preset continuous step size, and the internal activation response tensor is continuously flattened in physical memory to reconstruct a two-dimensional dedicated activation space topology matrix.

[0010] According to the large language model deployment method based on task awareness and hardware constraints provided by the present invention, in step S2, the two-dimensional dedicated activation space topology matrix in the unified analysis context is segmented to obtain cache adaptation sub-blocks, the cache adaptation sub-blocks are sequentially sent to the tensor core to perform block matrix multiplication to obtain the operation results, and the operation results are aggregated and accumulated on-chip to obtain the second-order covariance matrix.

[0011] According to the large language model deployment method based on task awareness and hardware constraints provided by the present invention, in step S2, a layer activation matrix is ​​obtained by uniformly analyzing the context, cross-layer error propagation coefficients are obtained based on the layer activation matrix, channel sensitivity scores are obtained through the cross-layer error propagation coefficients and the basic leverage score, and the set of the channel sensitivity scores is used as the global channel sensitivity score set.

[0012] According to the large language model deployment method based on task awareness and hardware constraints provided by the present invention, in step S3, the global channel sensitivity score set is input into the on-chip sorting accelerator, and the original channel index redirection pointer array is generated in descending order of absolute value. The data blocks are divided according to the original channel index redirection pointer array to obtain physical data blocks.

[0013] According to the large language model deployment method based on task awareness and hardware constraints provided by the present invention, in step S4, the physical parameters of the terminal device are retrieved from the hardware constraints, and the peak memory usage is calculated based on the physical parameters of the terminal device.

[0014] According to the large language model deployment method based on task awareness and hardware constraints provided by the present invention, in step S4, block precision is divided according to block-level candidate region threshold and block-level sensitivity to obtain initial block-level precision allocation result, memory usage parameters of the initial block-level precision allocation result are calculated, the remaining available video memory budget is calculated through the peak video memory usage, and the initial block-level precision allocation result is adjusted according to the remaining available video memory budget and memory usage parameters to obtain the block-level precision allocation result.

[0015] According to the task-aware and hardware-constrained large language model deployment method provided by the present invention, in step S5, a bit-width state mask is generated for the physical data block according to the block-level precision allocation result, the bit-width state mask is combined into a global precision mask table, and the maximum absolute value search is performed on the data within the physical data block with the physical data block as the smallest granularity to obtain the shared micro-scaling factor.

[0016] This invention also provides a large language model deployment system based on task awareness and hardware constraints, including: Unified Analysis Context Module: Used to acquire natural language text, generate probe data streams based on natural language text, use probe data streams to obtain an initial physical monitoring view and a two-dimensional dedicated activation space topology matrix, and obtain a unified analysis context through the initial physical monitoring view and the two-dimensional dedicated activation space topology matrix; The Global Channel Sensitivity Score Set module is used to construct a second-order covariance matrix based on a unified analysis context, calculate the noise tolerance constant of the second-order covariance matrix, calculate the basic leverage score based on the noise tolerance constant, and obtain the global channel sensitivity score set through the basic leverage score. Block-level sensitivity module: used to divide data into physical data blocks based on the global channel sensitivity score set, and to aggregate the channel sensitivity of the physical data blocks to obtain the block-level sensitivity; Block-level precision allocation module: It is used to obtain hardware constraints, calculate peak video memory usage based on hardware constraints, determine the threshold of block-level candidate regions, and perform block precision division based on block-level sensitivity, peak video memory usage and block-level candidate region thresholds to obtain block-level precision allocation results. Large Language Model Deployment Module: This module generates a global precision mask table based on the block-level precision allocation results, generates a shared micro-scaling factor, uses the global precision mask table and the shared micro-scaling factor to obtain a static deployment descriptor, and deploys the target model based on the static deployment descriptor to complete the large language model deployment.

[0017] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The present invention provides a method and system for deploying large language models based on task awareness and hardware constraints. By combining task awareness with hardware constraints, it utilizes a unified analysis context and a global channel sensitivity score set to achieve accurate data block partitioning and block-level sensitivity aggregation. Then, based on the peak memory usage and block-level candidate region threshold under hardware constraints, it completes block-level precision allocation. The generated global precision mask table and shared micro-scaling factor are used to construct a static deployment descriptor. While ensuring the quality of model deployment, it significantly reduces memory overhead and computational complexity, improves deployment efficiency and hardware adaptability, and achieves efficient and robust deployment of large language models.

[0018] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the large language model deployment method based on task awareness and hardware constraints provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the structure of the large language model deployment system based on task awareness and hardware constraints provided by the present invention.

[0022] Figure label: 100. Unified Analysis Context Module; 200. Global Channel Sensitivity Score Set Module; 300. Block-Level Sensitivity Module; 400. Block-Level Precision Allocation Module; 500. Large Language Model Deployment Module. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0024] In the description of the embodiments of the present invention, it should be noted that the terms "first", "second" and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0025] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0026] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0027] The following is combined Figures 1 to 2 Specific embodiments of the present invention are described below. Figure 1 A flowchart illustrating the large language model deployment method based on task awareness and hardware constraints provided by this invention includes: S1: Obtain natural language text, generate probe data stream based on natural language text, use probe data stream to obtain initial physical monitoring view and two-dimensional dedicated activation space topology matrix, and obtain unified analysis context through initial physical monitoring view and two-dimensional dedicated activation space topology matrix; Furthermore, the objective of this stage is to obtain a unified analysis context through the initial physical monitoring view and the two-dimensional dedicated activation space topology matrix. Specifically, in step S1, word segmentation and encoding processing is performed on the natural language text to obtain an input feature tensor, and the input feature tensor is loaded to obtain the probe data stream.

[0028] In step S1, the internal activation response tensor is obtained through the probe data stream, a preset continuous step size is determined, the direct memory access controller is scheduled to perform merged memory access according to the preset continuous step size, and the internal activation response tensor is continuously flattened in the physical video memory to reconstruct a two-dimensional dedicated activation space topology matrix.

[0029] The specific implementation method for the above steps in this embodiment is as follows: Here, the memory management unit of the edge device first needs to be activated to allocate a non-pageable high-speed video memory region for the static weight parameters of the large language model to be deployed, and directly map the original high-precision floating-point weights to a continuous physical address space to form an initial weight mapping matrix W. Then, natural language text corresponding to the target downstream task is acquired as probe data. Natural language text can include code text, question-and-answer text, instruction text, medical text, and in-vehicle scene interaction text. Word segmentation and encoding are then performed on the natural language text data to construct the model input feature tensor. The input feature tensor is loaded into a secondary cache or on-chip buffer to form a standardized probe data stream, thereby reducing statistical bias caused by fluctuations in the distribution of input data.

[0030] Subsequently, the large language model is configured to enter a gradient-free forward static inference mode. Driven by the probe data stream, the internal activation response tensor and layer activation matrix of the large language model are captured layer by layer. At the same time, the underlying hardware performance counters are called to collect the execution latency of the corresponding operators, the memory bandwidth utilization, the cache miss rate, and the tensor core utilization. The above content is spliced ​​together to construct an initial physical monitoring view that reflects the coupling relationship between task load and hardware bottleneck.

[0031] Given that activation responses generated during forward inference of large language models are typically multidimensionally distributed with batch, sequence, and channel dimensions, directly reading them in their original layout can lead to address jumps and memory fragmentation. Therefore, a preset continuous step size is determined, and the direct memory access controller is scheduled to perform merged memory accesses according to this preset continuous step size. Furthermore, the internal activation response tensor is continuously flattened in physical memory, reconstructing it into a two-dimensional dedicated activation space topology matrix. The two-dimensional dedicated activation space topology matrix. Initial weight mapping matrix and initial physical monitoring view The data is then spliced ​​together to form a unified analytical context for subsequent sensitivity decoding and constraint optimization.

[0032] S2: Construct a second-order covariance matrix based on the unified analysis context, calculate the noise tolerance constant of the second-order covariance matrix, calculate the basic leverage score based on the noise tolerance constant, and obtain the global channel sensitivity score set through the basic leverage score; Furthermore, the objective of this stage is to obtain a global channel sensitivity score set. Specifically, in step S2, the two-dimensional dedicated activation space topology matrix in the unified analysis context is segmented to obtain cache adaptation sub-blocks. The cache adaptation sub-blocks are then sequentially fed into the tensor core to perform block matrix multiplication to obtain the calculation results. The calculation results are then aggregated on-chip to obtain the second-order covariance matrix.

[0033] In step S2, the layer activation matrix is ​​obtained by unifying the analysis context, the cross-layer error propagation coefficient is obtained based on the layer activation matrix, the channel sensitivity score is obtained by the cross-layer error propagation coefficient and the basic leverage score, and the set of the channel sensitivity scores is used as the global channel sensitivity score set.

[0034] The specific implementation method for the above steps in this embodiment is as follows: For two-dimensional dedicated activation space topology matrix If the size of the data exceeds the capacity of the on-chip static random access memory (SRAM), it is divided into several cache adaptation sub-blocks along the sample dimension to fit the on-chip cache size. Each cache adaptation sub-block is then sequentially fed into the tensor core to perform block matrix multiplication, and the internal accumulation register is forced to FP32 high-precision mode to obtain the computation results. The computation results of all cache adaptation sub-blocks are then aggregated and accumulated on-chip to generate the second-order covariance matrix for the target task. The second-order covariance matrix is ​​used to characterize the correlation, energy distribution, and task feature coupling relationship between channels, providing second-order information support for key channel decoding.

[0035] The mean of the main diagonal elements of the second-order covariance matrix is ​​extracted as the feature scale reference, and a noise tolerance constant is defined. : in, The default minimum positive constant; Tr() extracts the mean of the main diagonal elements within the parentheses. Representing the matrix dimension. The noise tolerance constant is compared with the identity matrix. Superimposed, the underlying matrix inversion acceleration unit is called to generate the steady-state decoding matrix: This regularization operation can suppress matrix ill-conditioning caused by feature sparsity and avoid inversion anomalies caused by singular matrices.

[0036] Based on the second-order covariance matrix and its steady-state decoding matrix, the basic leverage score of the i-th physical channel is calculated. : Here, diag() means to extract the diagonal element within the parentheses.

[0037] Furthermore, each physical channel has a corresponding layer activation matrix, and the layer activation matrix corresponding to the i-th physical channel is the layer activation matrix of the i-th physical channel. The layer activation matrix, which can be obtained from the initial physical monitoring view of the unified analysis context. Extract the layer activation matrix. The Frobenius norm of the activation matrix of layer 1 is calculated by dividing it by the mean of the Frobenius norms of the activation matrices of all layers in the entire network, thus obtaining the first... Layer cross-layer error propagation coefficient This allows us to calculate the channel sensitivity score for the i-th physical channel. : The set of channel sensitivity scores for all physical channels is used as the global channel sensitivity score set.

[0038] S3: Divide the data into physical data blocks based on the global channel sensitivity score set, and aggregate the channel sensitivity of the physical data blocks to obtain the block-level sensitivity; Furthermore, the objective of this stage is to obtain block-level sensitivity. Specifically, in step S3, the global channel sensitivity score set is input into the on-chip sorting accelerator, and an original channel index redirection pointer array is generated in descending order of absolute value. Data blocks are then divided according to the original channel index redirection pointer array to obtain physical data blocks.

[0039] Set of channel sensitivity scores The input to the on-chip sorting accelerator generates an array of original channel index redirection pointers in descending order of absolute value. : The SortDesc() function sorts the contents within the parentheses in descending order. The length is .

[0040] Physical block alignment step size required by the micro-floating-point format Calculate the total number of physical blocks in the physical data block: in, Indicates rounding up, when Unable to be When performing integer division, several virtual indices pointing to zero placeholders can be appended to the end of the redirected pointer array.

[0041] The redirect pointer array is divided according to the total number of physical blocks. From a series of consecutive physical blocks, we obtain the k-th physical data block. .

[0042] For each physical data block The channel sensitivities within the block are aggregated to obtain the k-th block-level sensitivity. : in, This represents the number of physical channels in the k-th physical data block. Block-level sensitivity is used to characterize the overall importance of the entire physical data block during task execution.

[0043] S4: Obtain hardware constraints, calculate peak video memory usage based on hardware constraints, determine block-level candidate region thresholds, and perform block precision partitioning based on block-level sensitivity, peak video memory usage, and block-level candidate region thresholds to obtain block precision allocation results; Furthermore, the objective of this stage is to perform block-level precision partitioning to obtain block-level precision allocation results. Specifically, in step S4, the physical parameters of the end-side device are retrieved from the hardware constraints, and the peak memory usage is calculated based on the physical parameters of the end-side device.

[0044] In step S4, block precision is divided according to the block-level candidate region threshold and block-level sensitivity to obtain the initial block-level precision allocation result. The memory usage parameters of the initial block-level precision allocation result are calculated. The remaining available video memory budget is calculated through the peak video memory usage. The initial block-level precision allocation result is adjusted according to the remaining available video memory budget and memory usage parameters to obtain the block-level precision allocation result.

[0045] The specific implementation method for the above steps in this embodiment is as follows: The system interface is called to obtain the hardware constraints of the current edge device. The hardware constraints include edge physical parameters and hardware parameters, including the physical memory limit. Block alignment step size upper limit of bus bandwidth End-side physical parameters include batch size Maximum sequence length Number of attention heads With head dimension Based on the key-value cache growth pattern during autoregressive decoding of large language models, peak memory usage is dynamically calculated. : in, Indicates the length of a half-floating-point precision number.

[0046] Considering security fragment redundancy rate In this case, calculate the remaining available memory budget for static weights and activation tensors. : .

[0047] Set the data bit width to an 8-bit high-fidelity bit width, a 6-bit middle bit width, and a 4-bit low bit width, and set a threshold including the first block-level candidate region. Second block level candidate region threshold The block-level candidate region threshold is determined. The search step size is initialized to perform the threshold search in the block-level sensitivity distribution space.

[0048] Then, all physical data blocks are traversed according to the search step size. Next, high-fidelity bit widths are assigned to physical data blocks with block-level sensitivity higher than or equal to the threshold of the first block-level candidate region; intermediate bit widths are assigned to physical data blocks with block-level sensitivity lower than the threshold of the first block-level candidate region but higher than the threshold of the second block-level candidate region; and low bit widths are assigned to the rest, thus obtaining the initial block-level precision allocation result. This allows for the estimation of the weights and activation storage overhead of the current block-level bit width allocation scheme, yielding memory usage parameters. : in, Indicates the first bit width under high fidelity Storage overhead per physical data block, M Indicates the first bit width under the middle bit width Storage overhead per physical data block Indicates the first in the low bit width Storage overhead for each physical data block.

[0049] Based on the unified analysis context, the video memory bandwidth utilization and the initial block-level precision allocation results The length L of the different text input sequences that may occur during the inference phase can be used to estimate the bus bandwidth usage under the initial block-level precision allocation result, thus obtaining the bandwidth usage. Subsequently, if the current initial block-level precision allocation result can satisfy... and If these two conditions are met, the initial block-level precision allocation result will be used as the block-level precision allocation result Q. Otherwise, the threshold values ​​of the first and second block-level candidate regions need to be adjusted until the above conditions are met to obtain the block-level precision allocation result.

[0050] S5: Generate a global precision mask table based on the block-level precision allocation results, generate a shared micro-scaling factor, use the global precision mask table and the shared micro-scaling factor to obtain a static deployment descriptor, deploy the target model based on the static deployment descriptor, and complete the deployment of the large language model.

[0051] Furthermore, the objective of this stage is to deploy the target model and complete the deployment of the large language model. Specifically, in step S5, a bit-width state mask is generated for the physical data block based on the block-level precision allocation result. The bit-width state mask is combined into a global precision mask table, and a maximum absolute value search is performed on the data within the physical data block with the physical data block as the smallest granularity to obtain the shared micro-scaling factor.

[0052] The specific implementation method for the above steps in this embodiment is as follows: First, a bit-width status mask is generated for each physical data block based on the block-level precision allocation result. This bit-width status mask indicates the bit width assigned to each physical data block. The bit-width status masks are then combined into a global precision mask table. Then, using physical data blocks as the smallest granularity, a maximum absolute value search is performed on the data within each physical data block to obtain the shared micro-scaling factor of the k-th physical data block. Furthermore, it is necessary to determine the starting address of the k-th physical data block, thereby obtaining the starting address mapping relationship of the k-th physical data block. Furthermore, the addressing logic of the physical data blocks needs to be sent down to the direct memory access controller to construct a two-dimensional direct memory access transfer descriptor. This descriptor places the source address at the discrete physical address offset of the initial weight mapping matrix and the destination address at the address within the contiguous block. This drives the hardware to perform block-level collection operations, resulting in a rearranged weight matrix that satisfies the block-level contiguous layout. By concatenating the above content, you can obtain the static deployment descriptor. : Finally, this allows the client-side engine to directly parse the static deployment descriptor during the loading phase, enabling the deployment of the large language model as the target model on the client side based on the content of the static deployment descriptor, thus completing the deployment of the large language model.

[0053] The effectiveness of the task-aware and hardware-constrained large language model deployment method was also verified here. After deploying the large language model, the higher the accuracy of the large language model under the same task, the higher its deployment efficiency. Therefore, the accuracy of the same large language model deployed using different deployment methods to complete the same task can be used to measure the deployment efficiency of each method. The accuracy of the large language model deployed by our method and other methods under different tasks is shown in Table 1: Table 1. Accuracy of large language models deployed using this method and other methods on different tasks.

[0054] Table 1 shows the accuracy of large language models deployed using various methods when performing different tasks. Model 1 and Model 2 are existing large language models; FP16, QuaRot, QuaRot, Atom, AMXFP4, and Micromix are existing large language model deployment methods. Average bit width represents the average bit width of all data blocks in the large language model deployed using each method. 0-shot indicates no reference samples during task execution, 5-shot indicates 5 reference samples during task execution. ARC-C represents a high-difficulty scientific knowledge and reasoning task; BoolQ represents reading comprehension and true / false judgment tasks; Lambada represents text understanding and next-word prediction tasks; PIQA represents a physics commonsense reasoning task; WinoGrande represents commonsense reasoning and reference resolution tasks; and MMLU represents multidisciplinary knowledge and comprehensive reasoning tasks. It can be seen that our method achieves high accuracy with a relatively low average bit width across all tasks.

[0055] The following describes the large language model deployment system based on task awareness and hardware constraints provided by this invention. The large language model deployment system based on task awareness and hardware constraints described below and the large language model deployment method based on task awareness and hardware constraints described above can be referred to and correspond to each other.

[0056] Figure 2 An example is shown in the structural diagram of a large language model deployment system based on task awareness and hardware constraints, such as... Figure 2 As shown, the method for deploying large language models based on task awareness and hardware constraints as described above includes: Unified Analysis Context Module 100: Used to acquire natural language text, generate probe data stream based on natural language text, use probe data stream to obtain initial physical monitoring view and two-dimensional dedicated activation space topology matrix, and obtain unified analysis context through initial physical monitoring view and two-dimensional dedicated activation space topology matrix; Global Channel Sensitivity Score Set Module 200: This module is used to construct a second-order covariance matrix based on the unified analysis context, calculate the noise tolerance constant of the second-order covariance matrix, calculate the basic leverage score based on the noise tolerance constant, and obtain the global channel sensitivity score set through the basic leverage score. Block-level sensitivity module 300: Used to divide data into physical data blocks based on the global channel sensitivity score set, and to aggregate the channel sensitivity of the physical data blocks to obtain the block-level sensitivity; Block-level precision allocation module 400: It is used to obtain hardware constraints, calculate peak video memory usage based on hardware constraints, determine the threshold of block-level candidate regions, divide block precision based on block-level sensitivity, peak video memory usage and block-level candidate region threshold, and obtain block-level precision allocation results. Large Language Model Deployment Module 500: This module generates a global precision mask table based on the block-level precision allocation results, generates a shared micro-scaling factor, uses the global precision mask table and the shared micro-scaling factor to obtain a static deployment descriptor, and deploys the target model based on the static deployment descriptor to complete the large language model deployment.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for deploying large language models based on task awareness and hardware constraints, characterized in that: include: S1: Obtain natural language text, generate probe data stream based on natural language text, use probe data stream to obtain initial physical monitoring view and two-dimensional dedicated activation space topology matrix, and obtain unified analysis context through initial physical monitoring view and two-dimensional dedicated activation space topology matrix; S2: Construct a second-order covariance matrix based on the unified analysis context, calculate the noise tolerance constant of the second-order covariance matrix, calculate the basic leverage score based on the noise tolerance constant, and obtain the global channel sensitivity score set through the basic leverage score; S3: Divide the data into physical data blocks based on the global channel sensitivity score set, and aggregate the channel sensitivity of the physical data blocks to obtain the block-level sensitivity; S4: Obtain hardware constraints, calculate peak video memory usage based on hardware constraints, determine block-level candidate region thresholds, and perform block precision partitioning based on block-level sensitivity, peak video memory usage, and block-level candidate region thresholds to obtain block precision allocation results; S5: Generate a global precision mask table based on the block-level precision allocation results, generate a shared micro-scaling factor, use the global precision mask table and the shared micro-scaling factor to obtain a static deployment descriptor, deploy the target model based on the static deployment descriptor, and complete the deployment of the large language model.

2. The method for deploying a large language model based on task awareness and hardware constraints according to claim 1, characterized in that, In step S1, word segmentation and encoding are performed on the natural language text to obtain an input feature tensor. The input feature tensor is then loaded to obtain the probe data stream.

3. The method for deploying a large language model based on task awareness and hardware constraints according to claim 1, characterized in that, In step S1, the internal activation response tensor is obtained through the probe data stream, a preset continuous step size is determined, the direct memory access controller is scheduled to perform merged memory access according to the preset continuous step size, and the internal activation response tensor is continuously flattened in the physical video memory to reconstruct a two-dimensional dedicated activation space topology matrix.

4. The method for deploying a large language model based on task awareness and hardware constraints according to claim 1, characterized in that, In step S2, the two-dimensional exclusive activation space topology matrix in the unified analysis context is segmented to obtain cache adaptation sub-blocks. The cache adaptation sub-blocks are sequentially fed into the tensor core to perform block matrix multiplication to obtain the operation results. The operation results are then aggregated on-chip to obtain the second-order covariance matrix.

5. The method for deploying a large language model based on task awareness and hardware constraints according to claim 1, characterized in that, In step S2, the layer activation matrix is ​​obtained by unifying the analysis context, the cross-layer error propagation coefficient is obtained based on the layer activation matrix, the channel sensitivity score is obtained by the cross-layer error propagation coefficient and the basic leverage score, and the set of the channel sensitivity scores is used as the global channel sensitivity score set.

6. The method for deploying a large language model based on task awareness and hardware constraints according to claim 1, characterized in that, In step S3, the global channel sensitivity score set is input into the on-chip sorting accelerator, and the original channel index redirection pointer array is generated in descending order of absolute value. The data blocks are divided according to the original channel index redirection pointer array to obtain physical data blocks.

7. The method for deploying a large language model based on task awareness and hardware constraints according to claim 1, characterized in that, In step S4, the physical parameters of the terminal device are retrieved from the hardware constraints, and the peak video memory usage is calculated based on the physical parameters of the terminal device.

8. The method for deploying a large language model based on task awareness and hardware constraints according to claim 1, characterized in that, In step S4, block precision is divided according to the block-level candidate region threshold and block-level sensitivity to obtain the initial block-level precision allocation result. The memory usage parameters of the initial block-level precision allocation result are calculated. The remaining available video memory budget is calculated through the peak video memory usage. The initial block-level precision allocation result is adjusted according to the remaining available video memory budget and memory usage parameters to obtain the block-level precision allocation result.

9. The method for deploying a large language model based on task awareness and hardware constraints according to claim 1, characterized in that, In step S5, a bit-width state mask is generated for the physical data block according to the block-level precision allocation result. The bit-width state mask is combined into a global precision mask table. The maximum absolute value search is performed on the data within the physical data block with the physical data block as the smallest granularity to obtain the shared micro-scaling factor.

10. A large language model deployment system based on task awareness and hardware constraints, used to execute the large language model deployment method based on task awareness and hardware constraints as described in any one of claims 1 to 9, characterized in that, include: Unified Analysis Context Module: Used to acquire natural language text, generate probe data streams based on natural language text, use probe data streams to obtain an initial physical monitoring view and a two-dimensional dedicated activation space topology matrix, and obtain a unified analysis context through the initial physical monitoring view and the two-dimensional dedicated activation space topology matrix; The Global Channel Sensitivity Score Set module is used to construct a second-order covariance matrix based on a unified analysis context, calculate the noise tolerance constant of the second-order covariance matrix, calculate the basic leverage score based on the noise tolerance constant, and obtain the global channel sensitivity score set through the basic leverage score. Block-level sensitivity module: used to divide data into physical data blocks based on the global channel sensitivity score set, and to aggregate the channel sensitivity of the physical data blocks to obtain the block-level sensitivity; Block-level precision allocation module: It is used to obtain hardware constraints, calculate peak video memory usage based on hardware constraints, determine the threshold of block-level candidate regions, and perform block precision division based on block-level sensitivity, peak video memory usage and block-level candidate region thresholds to obtain block-level precision allocation results. Large Language Model Deployment Module: This module generates a global precision mask table based on the block-level precision allocation results, generates a shared micro-scaling factor, uses the global precision mask table and the shared micro-scaling factor to obtain a static deployment descriptor, and deploys the target model based on the static deployment descriptor to complete the large language model deployment.

Images