A model compression strategy optimization method for an intelligent middle platform integrated large model

Abstract

The application belongs to the technical field of artificial intelligence, and discloses a model compression strategy optimization method for an intelligent middle platform integrated large model; comprising: S1, obtaining a deployment environment constraint set of an original large model, inputting a sample calibration data set into the original large model for analysis, and generating a sensitivity configuration description file; S2, performing manifold projection and topological analysis on each sample feature, and constructing a task manifold index graph; S3, determining a search parameter, combining the sensitivity configuration description file and the task manifold index graph for logical association, and generating a virtual execution strategy; S4, performing feature projection on received actual input data based on the virtual execution strategy, and outputting a compressed reasoning result and a confidence index; S5, performing quality checking on the confidence index, if the checking is passed, then the compressed reasoning result is encapsulated, otherwise, full-precision re-reasoning of the original large model is triggered, so that a scientific and data-driven prediction basis is provided.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method for optimizing model compression strategies for intelligent middleware-integrated large models. Background Technology

[0002] With the acceleration of digital transformation, intelligent middleware, as a system platform supporting the service-oriented deployment and full lifecycle management of models, widely integrates large models to support complex businesses in industries such as finance and government. This intelligent middleware perceives business needs through task scheduling interfaces and service level agreements (SLAs), uniformly registers and calls models based on the model service catalog, and adapts computing power based on heterogeneous resource pools. However, large models have huge parameter scales and face latency and memory pressure in high-concurrency and resource-constrained scenarios, so an efficient compression and optimization solution is urgently needed.

[0003] Existing intelligent middleware compression models typically employ a static offline compression paradigm. This involves a serial processing flow of pruning, distillation, and quantization to transform a large model into a lightweight version with a fixed structure before deployment. This approach assumes that all input data have the same computational difficulty, ignoring the differences in the distribution of features on different input samples' manifolds and their sensitivity to model accuracy. This leads to rigid model reasoning logic. For example, in the question-and-answer scenario of an intelligent government middleware platform, users may alternate between submitting simple policy queries and complex legal interpretation requests. With existing static compression, for simple queries, the model cannot skip redundant deep calculations, resulting in excessive response latency during high-concurrency periods and significant waste of computing resources. For complex interpretations, because the model structure has been permanently reduced, key semantic reasoning capabilities are impaired, leading to logically confused output results or decreased accuracy. This inability to dynamically adapt computing power to input features traps the middleware platform in an efficiency dilemma.

[0004] In view of this, the present invention proposes a model compression strategy optimization method for intelligent middleware integrated large models to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned shortcomings of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a model compression strategy optimization method for an intelligent middleware integrated large model, comprising:

[0006] 1. A method for optimizing model compression strategies for a large integrated model in an intelligent middleware platform, characterized in that the method comprises:

[0007] S1. Obtain the deployment environment constraint set of the original large model, input the sample calibration dataset determined by matching the deployment environment constraint set into the original large model for analysis, and generate a sensitivity configuration description file.

[0008] S2. Perform manifold projection and topological analysis on the features of each sample corresponding to the sample calibration dataset to construct a task manifold index map containing the activation sparsity vectors of each network layer.

[0009] S3. In response to real-time business requests and service level agreement constraints, determine the retrieval parameters, and logically associate them with the sensitivity configuration description file and the task manifold index graph to generate a virtual execution strategy.

[0010] S4. Based on the virtual execution strategy, feature projection is performed on the received actual input data to locate and obtain neighboring nodes in the task manifold index graph. The activation sparsity vectors of the neighboring nodes are analyzed. Based on the layer scheduling mask and virtual quantization precision level determined by the analysis, lightweight compressed inference is performed on the original large model, and the compressed inference results and confidence index are output.

[0011] S5. Perform quality verification on the confidence index. If the verification passes, encapsulate the compressed inference results; otherwise, trigger full-precision re-inference of the original large model.

[0012] Furthermore, methods for inputting the sample calibration dataset determined based on the deployment environment constraint set matching into the original large model for analysis include:

[0013] The deployment environment constraint set of the original large model is obtained based on the task scheduling interface. The deployment environment constraint set includes business task type, service level agreement constraints and target hardware memory capacity limit.

[0014] Based on the business task types of the original large model, the pre-set task feature library is traversed to obtain a sample calibration dataset containing historical real business requests; the sample calibration dataset is input into the original large model to perform forward propagation operation to obtain the output loss function of the original large model, the sample features corresponding to the sample calibration dataset, and the output activation tensor of each sample feature in each network layer.

[0015] Backpropagation analysis is performed using the output loss function to obtain the gradients of the weight parameters of each network layer; the traces of the second derivative matrix of each network layer with respect to the output loss function are estimated based on the gradients; and the traces of the second derivative matrix are normalized to form the accuracy sensitivity index of each network layer.

[0016] Furthermore, methods for generating sensitivity configuration description files include:

[0017] The total memory budget of the original large model is determined based on the upper limit of the target hardware memory capacity; the total memory budget is proportionally allocated according to the accuracy sensitivity index of each network layer to determine the layer memory budget of each network layer.

[0018] Based on the memory budget of each layer, the maximum allowable quantization bit width is derived, and the maximum allowable quantization bit width is mapped to form the corresponding virtual quantization precision level; the reciprocal of the precision sensitivity index after normalization is calculated to determine the maximum tolerance pruning threshold of each network layer.

[0019] Extract the layer index numbers of each network layer in the original large model, associate and combine them with the corresponding virtual quantization accuracy level and the maximum tolerance pruning threshold to generate a sensitivity configuration description file.

[0020] Furthermore, methods for performing manifold projection and topological analysis on the features of each sample corresponding to the sample calibration dataset include:

[0021] Locality-sensitive hashing is performed on each sample feature to generate the corresponding hash code; density clustering analysis is performed on the sample features based on the Hamming distance of each hash code to form several clusters;

[0022] A preset activation threshold is set, and the proportion of elements whose output activation tensor absolute value is less than the activation threshold within the same cluster is counted to obtain the single-sample sparsity of the network layer corresponding to each sample feature. The average value of the single-sample sparsity of the corresponding network layer within the same cluster is calculated to determine the activation sparsity vector of each corresponding network layer.

[0023] Furthermore, methods for constructing a task manifold index graph containing the activation sparsity vectors of each network layer include:

[0024] The arithmetic mean of all sample features within each cluster is defined as a manifold node; the activation sparsity vector of each network layer is used as the attribute of the corresponding manifold node; the Euclidean distance between each manifold node is calculated, and the nearest neighbor node is selected for each manifold node based on the Euclidean distance and a connection edge is established to construct the task manifold index graph.

[0025] Furthermore, the methods for generating virtual execution strategies include:

[0026] The request service type tag is obtained by parsing the received real-time service requests, and the retrieval parameters are determined by combining the latency tolerance upper limit in the service level agreement constraints.

[0027] The virtual execution strategy is dynamically created. The virtual execution strategy includes query logic that performs a search for several nearest neighbor nodes in the task manifold index graph based on the search parameters, synthesis logic that synthesizes the target sparsity vector based on the attributes of the retrieved nearest neighbor nodes, and scheduling logic that generates a layer scheduling mask and virtual quantization precision level based on the target sparsity vector and sensitivity configuration description file.

[0028] Furthermore, the method for feature projection of the received actual input data based on the virtual execution strategy includes:

[0029] The actual input data is fed into the original large model to obtain semantic feature vectors and local sensitive hash codes. The query logic uses the query hash code as the index key and performs nearest neighbor search on the task manifold index graph through the retrieval parameters to locate and obtain several neighbor nodes of the actual input data and their similarity weights with the actual input data.

[0030] Furthermore, methods for analyzing the activation sparsity vectors of neighboring nodes include:

[0031] The synthesis logic performs a weighted average of the activation sparsity vectors of each neighboring manifold node based on the similarity weights of each neighboring node, generating a target sparsity vector that matches the actual input data.

[0032] The scheduling logic extracts the maximum tolerance pruning threshold for the corresponding network layer from the sensitivity configuration description file, and performs constraint verification on each element in the target sparsity vector with the corresponding maximum tolerance pruning threshold. If the element is greater than the preset sparsity decision threshold but does not exceed the maximum tolerance pruning threshold limit, a layer scheduling mask indicating that the corresponding network layer is in a skip state is generated; otherwise, a layer scheduling mask indicating that the corresponding network layer is in a computation state is generated. Based on the sensitivity configuration description file, the corresponding virtual quantization precision level is matched for the network layer marked as being in the computation state.

[0033] Furthermore, methods for performing lightweight compressed inference on the original large model include:

[0034] Perform data bypass operation on network layers marked as skipped; perform quantization tensor operation on network layers marked as computational state based on virtual quantization precision level to obtain compressed inference results;

[0035] The percentage of skipped states in the statistical layer scheduling mask; the average entropy of the output probability distribution of all computational state network layers; the reciprocal of the weighted sum of the percentage and the average entropy is determined as the confidence index of the compressed inference result.

[0036] Furthermore, methods for quality verification of confidence metrics include:

[0037] The confidence index is compared with the preset quality and safety threshold. If the confidence index is not less than the quality and safety threshold, the quality verification is deemed to have passed, and the compressed inference result is encapsulated and output.

[0038] If the confidence index is less than the quality and safety threshold, the quality verification is deemed to have failed, and a full-precision re-inference instruction is triggered. Based on the full-precision re-inference instruction, the actual input data is input into the original large model to perform forward propagation, obtain the corrected compressed inference result, and encapsulate and output it.

[0039] The actual input data and the corrected compressed inference results are stored as new samples in the task feature library.

[0040] The technical effects and advantages of the model compression strategy optimization method for intelligent middleware integrated large models of the present invention are as follows:

[0041] 1. This invention constructs a task manifold index graph, with manifold nodes as units and each network layer's activation sparsity vectors attached, by performing manifold projection and topological analysis on the sample features of historical calibration data. This structured the correspondence between massive discrete historical inputs and the model's internal behavior into a queryable empirical knowledge base. Through the intermediate representation of activation sparsity vectors, the computational redundancy of each layer within the original large model is explicitly recorded when encountering a certain type of input pattern. Simultaneously, manifold projection maps high-dimensional features to a low-dimensional hash space, significantly reducing the complexity of subsequent similarity matching and providing engineering feasibility for real-time input matching. This enables the system to understand the differences in the impact of different input data on the model's computational path, thus providing a scientific and data-driven prediction basis for subsequent dynamic invalid layer skipping based on layer scheduling masks, replacing blind empirical pruning.

[0042] 2. This invention determines retrieval parameters in response to real-time service requests and their service level agreement constraints. It logically associates static sensitivity configuration description files with dynamic task manifold index graphs to generate a virtual execution strategy encompassing query, synthesis, and scheduling logic. This strategy transforms the model compression mechanism from a fixed configuration into an adaptive process that can be adjusted in real-time according to the service quality requirements of each request. When ultra-fast response is required, the virtual execution strategy generates a more aggressive layer scheduling mask by adjusting retrieval parameters; when high precision is required, a conservative strategy is adopted. This achieves refined and personalized service quality assurance. Furthermore, by integrating the maximum tolerable pruning thresholds for each network layer specified in the sensitivity configuration description file and the virtual quantization accuracy level into the strategy generation, it ensures that any dynamic optimization will not exceed the safety boundaries determined by offline analysis. Thus, while pursuing inference performance, it fundamentally avoids the risk of model functional failure due to over-compression. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of a model compression strategy optimization method for an intelligent middleware integrated large model according to the present invention;

[0044] Figure 2 A schematic diagram illustrating the process of generating a sensitivity configuration description file for this invention;

[0045] Figure 3 This is a schematic diagram of the process for performing lightweight compressed inference in this invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Example 1, please refer to Figure 1 , Figure 2 and Figure 3 As shown in this embodiment, the main design contents of the model compression strategy optimization method for intelligent middleware integrated large models are as follows:

[0048] The inherent defects of existing static compression models in intelligent middleware service scenarios are that they use a unified and fixed computation path to process all inputs, completely ignoring the differences in the distribution of different input data in the feature manifold space and their differentiated needs for internal computational resources.

[0049] For example, in scenarios such as intelligent government Q&A, simple policy queries and complex legal interpretations are two tasks with vastly different semantic complexities. Existing technologies, through offline pruning, distillation, and quantization, produce static models whose network topology and computational precision are permanently fixed. This means that when processing simple queries, the model cannot dynamically skip redundant deep calculations for the current input, still executing unnecessary full computational graphs, resulting in wasted computing power and latency. Furthermore, when processing complex interpretations, because the key network layers supporting deep semantic understanding have been irreversibly reduced or over-quantized during compression, the model's high-order reasoning ability is "dimension-reduced," leading to chaotic output logic or a sharp drop in accuracy.

[0050] This rigid contradiction—that "low-complexity inputs cannot save computing power, while high-complexity inputs cannot maintain accuracy"—leads to a dual predicament for static compression models in dynamic business flows: low resource utilization and unstable service quality.

[0051] Based on this, a model compression strategy optimization method for intelligent middleware integrating large models is designed, including:

[0052] S1. Obtain the deployment environment constraint set of the original large model. Input the sample calibration dataset determined based on the deployment environment constraint set into the original large model for analysis, and generate a sensitivity configuration description file. Input the sample calibration dataset into the original large model, analyze and learn the accuracy sensitivity index of each network layer parameter in the original large model, and then, combined with hardware resource constraints, convert the accuracy sensitivity index into a specific compression safety boundary: determine the minimum compressible accuracy level (virtual quantization accuracy level) and the upper limit of activation sparsity that allows skipped calculations (maximum tolerance pruning threshold) for each network layer, and record them in the sensitivity configuration description file.

[0053] Methods for inputting a sample calibration dataset determined based on deployment environment constraint set matching into the original large model for analysis include:

[0054] The deployment environment constraint set of the original large model is obtained based on the task scheduling interface. This constraint set includes business task types, service level agreement (SLA) constraints, and the target hardware memory capacity limit. The intelligent platform sends a status query command to the task scheduling interface via a remote procedure call (RPC) protocol. The task scheduling interface responds to the query command and returns a JSON-formatted configuration file (containing key runtime parameters of the original large model in the current deployment scenario). The intelligent platform parses the JSON configuration file, extracting the task classification field values ​​and mapping them to business task types, the maximum response latency field values ​​and mapping them to SLA constraints, and the physical memory quota field values ​​and mapping them to the target hardware memory capacity limit.

[0055] For example, the configuration file content includes:

[0056] {"task_category":"text_generation","latency_limit_ms":100,"memory_cap_gb":16};

[0057] The intelligent middleware platform maps "task_category" to business task types;

[0058] Map "latency_limit_ms" to a service level agreement constraint;

[0059] Map "memory_cap_gb" to the maximum capacity of the target hardware's video memory.

[0060] Based on the business task types of the original large model, a pre-built task feature library is traversed to obtain a sample calibration dataset containing historical real business requests. The task feature library stores historical real business requests corresponding to various business task types of the original large model. The task feature library uses a key-value pair database structure, where the key is the task type label and the value is a list of historical real business requests. Using the business task type of the original large model as the matching key, a list of historical real business requests whose keys completely match the business task type is selected from the task feature library. The selected list of historical real business requests is then sorted in reverse order by storage timestamp, and a certain number (no less than 500, which can be selected according to the compression and optimization scenario of the original large model) of the latest historical real business requests are extracted from the list to form a sample calibration dataset matching the original large model.

[0061] The sample calibration dataset is input into the original large model to perform forward propagation. This yields the output loss function of the original large model, the features of each sample corresponding to the sample calibration dataset, and the output activation tensors of each sample feature in each network layer. The sample calibration dataset is divided into batches and input into the original large model in batch order (keeping it read-only). For each input batch, a complete forward propagation computation of the original large model is performed. During each forward propagation operation of the original large model, the following are recorded: the predicted probability distribution generated by the task output layer of the original large model; the output feature tensor generated by the specified intermediate feature layer of the original large model; and the output activation tensor of each network layer of the original large model (a multidimensional array composed of the output values ​​of all neurons in the corresponding network layer). The output of the specified feature extraction layer (which can be the same as the intermediate feature layer) for each sample is extracted and flattened into a one-dimensional vector, serving as the sample feature of the corresponding sample.

[0062] Calculate the cross-entropy loss between the predicted probability distribution of the task output layer and the true label; calculate the mean squared error loss between the output feature tensor of the intermediate feature layer and the reference feature tensor. Then, weight the cross-entropy loss and the mean squared error loss (weight coefficients are determined based on the task type) to obtain the output loss function.

[0063] It should be explained that the output loss function is used to quantify the degree of "imperfection" of the original large model under the current input. The larger the value of the output loss function, the greater the gap between the output of the original large model and the ideal state.

[0064] Backpropagation analysis is performed using the output loss function to obtain the gradients of the weight parameters of each network layer. Starting from the output loss function, during the backpropagation calculation, the derivatives of the weight parameters of each network layer in the original large model are calculated to obtain the gradient corresponding to each network layer weight parameter.

[0065] The trace value of the second derivative matrix of each network layer with respect to the output loss function is estimated based on gradient. For each network layer, a random vector is introduced using the gradient of that network layer and a second backpropagation is performed to directly estimate the second derivative matrix of that network layer. The trace value of the second derivative matrix (i.e., the sum of all elements on the main diagonal of the second derivative matrix) is extracted, and the trace values ​​of the second derivative matrix of each network layer are classified and stored according to the network layer index number of the original large model.

[0066] The trace values ​​of the second derivative matrix are normalized to form the accuracy sensitivity index for each network layer. The trace values ​​of the second derivative matrix of all network layers are traversed to obtain the maximum and minimum values. Using the minimum value as the baseline and the maximum value as the upper limit, a linear scaling process is performed on the trace values ​​of the second derivative matrix of each network layer (uniformly mapping them to the numerical range of 0 to 1). The resulting trace values ​​of the second derivative matrix of each network layer are the corresponding accuracy sensitivity indexes.

[0067] For example, the original large model has 3 layers, and the calculated trace values ​​of the second derivative matrices are 5.0, 1.0, and 3.0, respectively. The maximum trace value is 5.0, and the minimum trace value is 1.0.

[0068] The accuracy sensitivity index for the first layer is (5.0-1.0)÷(5.0-1.0)=1.0; the accuracy sensitivity index for the second layer is (1.0-1.0)÷(5.0-1.0)=0.0; and the accuracy sensitivity index for the third layer is (3.0-1.0)÷(5.0-1.0)=0.5.

[0069] Methods for generating sensitivity configuration description files include:

[0070] The total memory budget for the original large model is determined based on the target hardware memory capacity limit. The total memory budget is then proportionally allocated to each network layer using its precision sensitivity index, thus determining the layer memory budget for each network layer. The target hardware memory capacity limit is multiplied by a buffer coefficient (typically between 0.7 and 0.9, used to reserve redundant memory space for caching intermediate activation values ​​and key values ​​during inference) to obtain the total memory budget. The sum of the precision sensitivity indices of all network layers is used as the overall proportional baseline. The memory allocation ratio for each network layer is obtained by dividing its precision sensitivity index by the sum of all network layer precision sensitivity indices. Finally, the memory allocation ratio is multiplied by the total memory budget to determine the layer memory budget for each network layer.

[0071] The maximum allowable quantization bit width is derived based on the memory budget of each layer, and then mapped to the corresponding virtual quantization precision level. The number of weight parameters in each network layer of the original large model is counted. The memory budget of each network layer is divided by the number of weight parameters, and the quotient is multiplied by the bit conversion constant 8 to calculate the theoretical average bit width of each network layer. The theoretical average bit width of each network layer is compared with the standard quantization bit width set. Integer bit width values ​​smaller than and closest to the theoretical average bit width are selected from the standard quantization bit width set and determined as the maximum allowable quantization bit width. A mapping table between the maximum allowable quantization bit width and the virtual quantization precision level is established. The enumerated value corresponding to the maximum allowable quantization bit width is found in the mapping table, and the found enumerated value is mapped to the corresponding virtual quantization precision level.

[0072] For example, the standard quantization bit width set includes 32, 16, 8, and 4. If the theoretical average bit width is 6.5, then 4 is selected as the result.

[0073] For example, a bit width value of 8 is mapped to the enumeration value INT8, and a bit width value of 4 is mapped to the enumeration value INT4;

[0074] An 8-bit quantization width corresponds to a high virtual quantization precision level, a 4-bit quantization width corresponds to a medium virtual quantization precision level, a 2-bit quantization width corresponds to a low virtual quantization precision level, and a 1-bit quantization width corresponds to an extremely low virtual quantization precision level.

[0075] Calculate the reciprocal of the accuracy sensitivity index after normalization to determine the maximum tolerance pruning threshold for each network layer.

[0076] It should be explained that if the reciprocal of the accuracy sensitivity index is 0, then its reciprocal is defined as 1.0; if the reciprocal of the accuracy sensitivity index is greater than 1.0, then the maximum tolerance pruning threshold is set to 1.0.

[0077] Extract the layer index numbers of each network layer in the original large model, associate and combine them with the corresponding virtual quantization accuracy level and the maximum tolerance pruning threshold to generate a sensitivity configuration description file.

[0078] S2. Perform manifold projection and topological analysis on the features of each sample corresponding to the sample calibration dataset to construct a task manifold index map containing activation sparsity vectors of each network layer. The sample features in the sample calibration dataset are mapped to a low-dimensional space through manifold projection (locality-sensitive hashing) and clustered to form "manifold nodes" representing different input patterns. For each manifold node, the activation sparsity of its corresponding samples in each layer of the original large model is statistically analyzed, and the average value is calculated to form the activation sparsity vector of each network layer of that manifold node. This constructs the task manifold index map, recording the historical experience of computational redundancy of each network layer of the original large model under different input patterns (manifold nodes). This provides a data foundation for subsequent real-time matching and prediction.

[0079] Methods for performing manifold projection and topological analysis on the features of each sample corresponding to the sample calibration dataset include:

[0080] Locality-sensitive hashing (LSH) is performed on each sample feature to generate the corresponding hash code. A random projection matrix following a Gaussian distribution is initialized, with its dimension matching the dimension of each sample feature. Matrix multiplication is performed between each sample feature and the random projection matrix. The projection result is then binarized using a sign function to convert continuous values ​​into binary bit sequences. These binary bit sequences are then concatenated to generate the corresponding hash code.

[0081] Density clustering analysis is performed on sample features based on the Hamming distance of each hash code to form several clusters. The hash codes corresponding to each sample feature are extracted, and the number of bits that differ between any two hash codes is counted; this number represents the Hamming distance between the two sample features. The Hamming distance between all sample features is calculated. The operating parameters of the density-based clustering algorithm are configured (the cluster radius parameter is configured as the distance threshold for neighborhood determination, and the minimum number of samples in a cluster is set as the density threshold for core point determination). For each sample feature, other sample features with a Hamming distance less than or equal to the cluster radius parameter are retrieved and marked as neighborhood samples of the current sample feature. The total number of neighborhood samples is counted; if the total number is greater than or equal to the minimum number of samples in a cluster, the current sample feature is determined to meet the density requirement, forming a cluster; otherwise, it is discarded.

[0082] For example, if the cluster radius is set to 3 and the minimum number of samples in a cluster is 5, if sample feature A has 6 neighboring samples within a radius of 3 (i.e., the total number is 6), then 6 is greater than 5, and sample feature A and its neighbors are identified as a valid cluster; if sample feature C has only 2 neighboring samples within a radius of 3, then 2 is less than 5, and sample feature C is judged as invalid noise and removed.

[0083] A preset activation threshold is set, and the proportion of elements within the same cluster whose absolute value of the output activation tensor is less than the threshold is counted to obtain the single-sample sparsity of the network layer corresponding to each sample feature. The magnitude of the single-sample sparsity is used to quantify the computational load characteristics of the network layer. If the single-sample sparsity is closer to 1, it indicates that the vast majority of elements in the output activation tensor are non-activated elements, and the corresponding network layer is determined to be in a state of high computational redundancy. If the single-sample sparsity is closer to 0, it indicates that the majority of elements in the output activation tensor are activated elements, and the corresponding network layer is determined to be in a state of dense feature information, and is identified as a critical computation path.

[0084] Calculate the average sparsity of a single sample in the corresponding network layer within the same cluster, and determine the activation sparsity vector of each corresponding network layer.

[0085] It should be explained that the activation threshold is used to determine the validity of each element in the output activation tensor and to distinguish between activated and non-activated elements. The activation threshold is set based on the statistical analysis of the distribution of output activation values ​​of each layer of the original large model on the sample calibration dataset.

[0086] Methods for constructing a task manifold index graph containing the activation sparsity vectors of each network layer include:

[0087] The arithmetic mean of all sample features within each cluster is defined as a manifold node; the activation sparsity vector of each network layer is used as the attribute of the corresponding manifold node; the Euclidean distance between each manifold node is calculated, and the nearest neighbor node is selected for each manifold node based on the Euclidean distance and a connection edge is established to construct the task manifold index graph.

[0088] S3. Responding to real-time business requests and service level agreement constraints, the retrieval parameters are determined. Logical association is then performed between the sensitivity configuration description file and the task manifold index graph to generate a virtual execution strategy. For real-time business requests, a dynamic and detailed execution plan (virtual execution strategy) that guides efficient reasoning is generated, defining a set of behavioral logics for "how to query the graph," "how to synthesize decisions," and "how to schedule computation."

[0089] Methods for generating virtual execution strategies include:

[0090] The system parses the received real-time business requests to obtain the request business type tag, and derives the retrieval parameters by combining this tag with the latency tolerance cap in the service level agreement (SLA) constraints. It then parses the protocol header of the real-time business requests to extract the business type field as the request business type tag. Finally, it selects the specific value of the latency tolerance cap from the SLA constraints in the deployment environment constraint set, using this value as the core constraint benchmark. Based on the latency tolerance cap, it performs a full-process time budget breakdown (e.g., allocating 50% of the latency tolerance cap as the time quota for the retrieval stage, and the remaining 50% as the reserved quota for the synthesis and scheduling stages).

[0091] The initial search parameters (including the number of nearest neighbor nodes retrieved, the precision of Local Sensitive Hash Projection, and the search range threshold) are derived based on the time-consuming quota of the search process. The derivation logic is configured as follows: if the time-consuming quota value increases, the number of nearest neighbor nodes retrieved and the precision of Local Sensitive Hash Projection are increased proportionally. Fine-tuning is then performed on the initial search parameters based on the request service type tag: if the request service type tag indicates a high-frequency, lightweight service, the number of nearest neighbor nodes retrieved and the precision of Local Sensitive Hash Projection are adjusted downwards; if it indicates a high-precision service, they are adjusted upwards. After the adjustment is completed, the final search parameters are locked.

[0092] For example, the latency tolerance limit is 100ms, the retrieval quota is 50ms, and the initial derivation is 5 nearest neighbor nodes to be retrieved and 64-bit projection precision. Due to the business tag being "intelligent text generation" (high frequency and lightweight), it is finely adjusted to: 3 nearest neighbor nodes to be retrieved, 32-bit projection precision, and Euclidean distance ≤ 2.0 for the retrieval range threshold.

[0093] The virtual execution strategy is dynamically created. The virtual execution strategy includes query logic that performs a search for several nearest neighbor nodes in the task manifold index graph based on the search parameters, synthesis logic that synthesizes the target sparsity vector based on the attributes of the retrieved nearest neighbor nodes, and scheduling logic that generates a layer scheduling mask and virtual quantization precision level based on the target sparsity vector and sensitivity configuration description file.

[0094] S4. Based on the virtual execution strategy, feature projection is performed on the received actual input data. (Through query logic) the neighbor nodes in the task manifold index graph are located and obtained. (By calling the synthesis logic and scheduling logic) the activation sparsity vectors of the neighbor nodes are analyzed. Based on the layer scheduling mask (predicted sparsity of each layer) and virtual quantization precision level (matching the precision level of the network layer to be calculated) determined by the analysis, lightweight compressed inference is performed on the original large model, and the compressed inference results and confidence index are output.

[0095] Methods for feature projection of received actual input data based on virtual execution strategies include:

[0096] The actual input data is fed into the original large model to obtain semantic feature vectors and locality-sensitive hash codes. The actual input data is the original business input data received by the intelligent platform, corresponding one-to-one with real-time business requests. It is the direct input object for the original large model to perform lightweight inference. (For example, if the request business type tag for a real-time business request is "intelligent text generation," then the corresponding actual input data is text-based request data). After preprocessing, the actual input data is input into the original large model on a single data point basis, initiating the feature extraction process of the original large model (only the feature extraction stage of the forward propagation operation of the original large model is executed). In the feature extraction process, the high-dimensional vector output by the feature extraction layer of the original large model is extracted (i.e., the semantic feature vector corresponding to the actual input data). Locality-sensitive hashing is performed on the semantic feature vector, and the parameters used in the operation are consistent with those used when processing sample features.

[0097] For example, the semantic feature vector corresponding to the actual input data is 1024-dimensional. After performing locality-sensitive hashing operations through 128 hash functions, a 128-bit binary code "1010...0110" is generated. This code is the locality-sensitive hash code corresponding to the actual input data.

[0098] The query logic uses the query hash code as the index key and performs a nearest neighbor search on the task manifold index graph through retrieval parameters to locate and obtain several neighbor nodes of the actual input data and their similarity weights. Based on the retrieval range threshold (Euclidean distance corresponding to Hamming distance threshold) in the retrieval parameters, manifold nodes with a Hamming distance less than or equal to the retrieval range threshold are selected, and then sorted in ascending order of Hamming distance values. Based on the number of nearest neighbor nodes retrieved in the retrieval parameters, several neighbor nodes corresponding to the sorted manifold nodes are selected, and the node number and corresponding attributes of each neighbor node are recorded.

[0099] For each selected neighbor node, calculate the Euclidean distance between the average vector corresponding to that neighbor node and the semantic feature vector corresponding to the actual input data; take the reciprocal of the Euclidean distance and normalize it to obtain the similarity weight between that neighbor node and the actual input data.

[0100] Methods for analyzing the activation sparsity vectors of neighboring nodes include:

[0101] The synthesis logic uses a weighted average of the activation sparsity vectors of each neighboring manifold node based on the similarity weights of each neighboring node to generate a target sparsity vector that matches the actual input data. It reads the attributes of each neighboring node one by one, and filters out the activation sparsity of each network layer corresponding to each neighboring node from the attributes; it then associates the similarity weights of each neighboring node with the corresponding activation sparsity. For each network layer of the original large model: it calculates the product of the activation sparsity of all neighboring nodes corresponding to that network layer with their corresponding similarity weights, and then sums all the products to obtain the target sparsity value of that network layer. The target sparsity values ​​of all network layers are arranged sequentially according to the network layer index numbers of the original large model, forming a vector consistent with the number of network layers.

[0102] The scheduling logic extracts the maximum tolerable pruning threshold for the corresponding network layer from the sensitivity configuration description file. It then performs a constraint check on each element in the target sparsity vector against the corresponding maximum tolerable pruning threshold. If an element is greater than the preset sparsity decision threshold but does not exceed the maximum tolerable pruning threshold limit, a layer scheduling mask indicating that the corresponding network layer is in a skipped state is generated. Otherwise, a layer scheduling mask indicating that the corresponding network layer is in a computation state is generated. If the constraint check is satisfied, the scheduling logic generates a flag with a value of 0, indicating that the corresponding network layer is in a skipped state, and writes this flag to the corresponding index position of the layer scheduling mask.

[0103] For example, the prediction sparsity of the 5th layer of the network is 0.68, the prediction sparsity of the 10th layer is 0.8, the sparsity decision threshold is 0.7, and the maximum tolerance pruning threshold is 0.9.

[0104] The condition 0.68 > 0.7 is false. Therefore, the scheduling mask for layer 5 is set to 0, and it is marked as "computation state". The condition (0.8 > 0.7) and (0.8 ≤ 0.9) are true. Therefore, the scheduling mask for layer 10 is set to 1, and it is marked as "skip state".

[0105] Based on the sensitivity configuration description file, the corresponding virtual quantization precision level is matched to the network layer marked as the computation state.

[0106] It should be explained that the sparse decision threshold is jointly derived and set based on the statistical distribution of activation sparsity of the original large model on the sample calibration dataset and the upper limit of latency tolerance of the service level protocol in the deployment environment.

[0107] S5. Perform quality verification on the confidence index. If the verification passes, encapsulate the compressed inference results; otherwise, trigger full-precision re-inference of the original large model.

[0108] Methods for performing lightweight compressed inference on the original large model include:

[0109] For network layers marked as skipped, a data bypass operation is performed. The data bypass operation involves: for each skipped network layer, blocking all matrix multiplications and nonlinear activation calculations within that layer, directly reading the input tensor of that layer (i.e., the output tensor of the previous layer), losslessly copying it, and using it as the output tensor of the current network layer, directly passing it to the next layer of the original large model. This utilizes the residual connection property to maintain the coherence and dimensionality consistency of the data flow, thereby eliminating the computational time consumption of that network layer.

[0110] For example, if the network layer with level index 3 is marked as skipped, then the output tensor with level index 2 is directly passed to the network layer with level index 4 as input.

[0111] Based on the virtual quantization precision level, quantization tensor operations are performed on network layers marked as computational states to obtain compressed inference results. For each network layer marked as computational state: the virtual quantization precision level associated with the network layer is read, and the corresponding low-precision operator kernel is called; the input tensor and weight parameters of the network layer are dynamically mapped to the discrete integer value range limited by the virtual quantization precision level; tensor convolution or matrix multiplication operations are performed in low-bit mode; and the operation results are dequantized back to floating-point values ​​to obtain the output tensor of the network layer.

[0112] Following the hierarchical topology of the original large model, the above-mentioned data bypass operation and quantization tensor operation are executed alternately until the data stream propagates to the last layer of the original large model; the output tensor of the last layer of the original large model is extracted, and decoding and word embedding processing are performed to convert it into structured data that conforms to the semantics of real-time business requests, forming compressed inference results.

[0113] The percentage of skipped states in the statistical layer scheduling mask. The number of network layers marked as skipped states in the statistical layer scheduling mask; the percentage of skipped states is calculated as the ratio of the number of skipped network layers to the total number of network layers in the original large model.

[0114] Calculate the average entropy of the output probability distributions of all computational state network layers. Extract the output tensors of all network layers labeled as computational states, and perform probability distribution calculations on the output tensors of each computational state network layer to obtain the output probability distribution of each computational state network layer. Calculate the information entropy of each output probability distribution vector to obtain the single-layer entropy value of each computational state network layer. Perform an arithmetic mean operation on the single-layer entropy values ​​of all computational state network layers to obtain the average entropy value.

[0115] The reciprocal of the weighted sum of the proportion and the average entropy value is determined as the confidence index of the compressed inference result.

[0116] It should be explained that the weighting coefficients of the proportion and average entropy value are adaptively configured based on the request business type label. If the real-time task request is a task with strong logical dependencies (which has extremely high requirements for the integrity of the deep inference chain of the original large model), the weighting coefficient of the proportion is increased to strictly control the risks caused by excessive pruning; if the real-time business request is a highly divergent task (the difficulty lies in the ambiguity of the input semantics and the divergence of the generation space), the weighting coefficient of the average entropy value is increased to keenly capture the "confusion" state when the original large model makes predictions.

[0117] Methods for quality verification of confidence metrics include:

[0118] The confidence index is compared with a preset quality and safety threshold. If the confidence index is not less than the quality and safety threshold, the quality verification is deemed successful (this compressed inference is valid), and the compressed inference result is encapsulated and output. The compressed inference result is encapsulated as a standard service interface, and external access is granted through the model service catalog of the intelligent platform.

[0119] Simultaneously, the virtual execution strategy parameters used in this inference (including the layer scheduling mask and virtual quantization precision level) and the calculated confidence index are extracted and stored as evaluation results for subsequent strategy recommendation and adaptive initialization reference when processing similar business tasks.

[0120] It should be explained that the quality and safety threshold is based on the minimum accuracy requirement specified in the service level agreement. It is determined by plotting a confidence-accuracy calibration curve on the historical validation set and finding the confidence threshold that meets the minimum accuracy requirement, so as to ensure that only compressed inference results with a high probability of being correct are released.

[0121] If the confidence index is less than the quality and safety threshold, the quality verification is deemed to have failed (the current virtual execution strategy is revoked), and a full-precision re-inference instruction is triggered. Based on the full-precision re-inference instruction, the actual input data is input into the original large model to perform forward propagation (full FP16 or FP32 precision), all network layers are traversed for lossless computation, the corrected compressed inference result is obtained, and it is encapsulated and output.

[0122] The actual input data and the corrected compressed inference results are stored as new samples in the task feature library. The actual input data that triggers re-inference and the corrected inference results are combined to construct new sample pairs. The business type label of the actual input data is extracted as the index key, and the sample pairs are written into the task feature library.

[0123] In this embodiment, by performing manifold projection and topological analysis on the sample features of historical calibration data, a task manifold index graph is constructed, with manifold nodes as units and each network layer's activation sparsity vectors attached. This structures the correspondence between massive discrete historical inputs and the model's internal behavior into a queryable empirical knowledge base. Through the intermediate representation of activation sparsity vectors, the computational redundancy of each layer within the original large model is explicitly recorded when encountering a certain type of input pattern. Simultaneously, by mapping high-dimensional features to a low-dimensional hash space through manifold projection, the complexity of subsequent similarity matching is significantly reduced, providing engineering feasibility for real-time input matching. This enables the system to understand the differences in the impact of different input data on the model's computational path, thus providing a scientific and data-driven predictive basis for subsequent dynamic invalid layer skipping based on layer scheduling masks, replacing blind empirical pruning.

[0124] By responding to real-time business requests and their service level agreement constraints to determine retrieval parameters, and logically associating static sensitivity configuration description files with dynamic task manifold index graphs, a virtual execution strategy is generated that includes query, synthesis, and scheduling logic. This strategy transforms the model compression mechanism from a fixed configuration to an adaptive process that can be adjusted in real time according to the service quality requirements of each request. When ultra-fast response is required, the virtual execution strategy generates a more aggressive layer scheduling mask by adjusting retrieval parameters; when high precision is required, a conservative strategy is adopted. This achieves refined and personalized service quality assurance. At the same time, by incorporating the maximum tolerable pruning thresholds for each network layer and the virtual quantization precision level specified in the sensitivity configuration description file into the strategy generation, it is ensured that any dynamic optimization will not exceed the security boundaries determined by offline analysis. Thus, while pursuing inference performance, the risk of model functional failure due to over-compression is fundamentally avoided.

[0125] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0126] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only one method, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0127] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0128] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for optimizing model compression strategies for intelligent middleware-integrated large models, characterized in that, The aforementioned model compression strategy optimization method for an integrated large model in an intelligent middleware platform includes: S1. Obtain the deployment environment constraint set of the original large model, input the sample calibration dataset determined by matching the deployment environment constraint set into the original large model for analysis, and generate a sensitivity configuration description file. S2. Perform manifold projection and topological analysis on the features of each sample corresponding to the sample calibration dataset to construct a task manifold index map containing the activation sparsity vectors of each network layer. S3. In response to real-time business requests and service level agreement constraints, determine the retrieval parameters, and logically associate them with the sensitivity configuration description file and the task manifold index graph to generate a virtual execution strategy. S4. Based on the virtual execution strategy, feature projection is performed on the received actual input data to locate and obtain neighboring nodes in the task manifold index graph. The activation sparsity vectors of the neighboring nodes are analyzed. Based on the layer scheduling mask and virtual quantization precision level determined by the analysis, lightweight compressed inference is performed on the original large model, and the compressed inference results and confidence index are output. S5. Perform quality verification on the confidence index. If the verification passes, encapsulate the compressed inference results; otherwise, trigger full-precision re-inference of the original large model.

2. The model compression strategy optimization method for an integrated large model in an intelligent middleware platform according to claim 1, characterized in that, The method for inputting the sample calibration dataset determined based on the deployment environment constraint set matching into the original large model for analysis includes: The deployment environment constraint set of the original large model is obtained based on the task scheduling interface. The deployment environment constraint set includes business task type, service level agreement constraints and target hardware memory capacity limit. Based on the business task types of the original large model, the pre-set task feature library is traversed to obtain a sample calibration dataset containing historical real business requests; the sample calibration dataset is input into the original large model to perform forward propagation operation to obtain the output loss function of the original large model, the sample features corresponding to the sample calibration dataset, and the output activation tensor of each sample feature in each network layer. Backpropagation analysis is performed using the output loss function to obtain the gradients of the weight parameters of each network layer; the traces of the second derivative matrix of each network layer with respect to the output loss function are estimated based on the gradients; and the traces of the second derivative matrix are normalized to form the accuracy sensitivity index of each network layer.

3. The method for optimizing model compression strategy of a large integrated model in an intelligent middleware platform according to claim 2, characterized in that, The method for generating the sensitivity configuration description file includes: The total memory budget of the original large model is determined based on the upper limit of the target hardware memory capacity; the total memory budget is proportionally allocated according to the accuracy sensitivity index of each network layer to determine the layer memory budget of each network layer. Based on the memory budget of each layer, the maximum allowable quantization bit width is derived, and the maximum allowable quantization bit width is mapped to form the corresponding virtual quantization precision level; the reciprocal of the precision sensitivity index after normalization is calculated to determine the maximum tolerance pruning threshold of each network layer. Extract the layer index numbers of each network layer in the original large model, associate and combine them with the corresponding virtual quantization accuracy level and the maximum tolerance pruning threshold to generate a sensitivity configuration description file.

4. The model compression strategy optimization method for an intelligent middleware integrated large model according to claim 3, characterized in that, The method for performing manifold projection and topological analysis on the features of each sample corresponding to the sample calibration dataset includes: Locality-sensitive hashing is performed on each sample feature to generate the corresponding hash code; density clustering analysis is performed on the sample features based on the Hamming distance of each hash code to form several clusters; A preset activation threshold is set, and the proportion of elements whose output activation tensor absolute value is less than the activation threshold within the same cluster is counted to obtain the single-sample sparsity of the network layer corresponding to each sample feature. The average value of the single-sample sparsity of the corresponding network layer within the same cluster is calculated to determine the activation sparsity vector of each corresponding network layer.

5. The model compression strategy optimization method for an intelligent middleware integrated large model according to claim 4, characterized in that, The method for constructing a task manifold index graph containing the activation sparsity vectors of each network layer includes: The arithmetic mean of all sample features within each cluster is defined as a manifold node; the activation sparsity vector of each network layer is used as the attribute of the corresponding manifold node; the Euclidean distance between each manifold node is calculated, and the nearest neighbor node is selected for each manifold node based on the Euclidean distance and a connection edge is established to construct the task manifold index graph.

6. The method for optimizing model compression strategy of a large integrated model in an intelligent middleware platform according to claim 5, characterized in that, The method for generating virtual execution strategies includes: The request service type tag is obtained by parsing the received real-time service requests, and the retrieval parameters are determined by combining the latency tolerance upper limit in the service level agreement constraints. The virtual execution strategy is dynamically created. The virtual execution strategy includes query logic that performs a search for several nearest neighbor nodes in the task manifold index graph based on the search parameters, synthesis logic that synthesizes the target sparsity vector based on the attributes of the retrieved nearest neighbor nodes, and scheduling logic that generates a layer scheduling mask and virtual quantization precision level based on the target sparsity vector and sensitivity configuration description file.

7. The method for optimizing model compression strategy of a large integrated model in an intelligent middleware platform according to claim 6, characterized in that, The method for feature projection of received actual input data based on a virtual execution strategy includes: The actual input data is fed into the original large model to obtain semantic feature vectors and local sensitive hash codes. The query logic uses the query hash code as the index key and performs nearest neighbor search on the task manifold index graph through the retrieval parameters to locate and obtain several neighbor nodes of the actual input data and their similarity weights with the actual input data.

8. The method for optimizing model compression strategy of an intelligent middleware integrated large model according to claim 7, characterized in that, The method for analyzing the activation sparsity vectors of neighboring nodes includes: The synthesis logic performs a weighted average of the activation sparsity vectors of each neighboring manifold node based on the similarity weights of each neighboring node, generating a target sparsity vector that matches the actual input data. The scheduling logic extracts the maximum tolerance pruning threshold for the corresponding network layer from the sensitivity configuration description file, and performs constraint verification on each element in the target sparsity vector with the corresponding maximum tolerance pruning threshold. If the element is greater than the preset sparsity decision threshold but does not exceed the maximum tolerance pruning threshold limit, a layer scheduling mask indicating that the corresponding network layer is in a skip state is generated; otherwise, a layer scheduling mask indicating that the corresponding network layer is in a computation state is generated. Based on the sensitivity configuration description file, the corresponding virtual quantization precision level is matched for the network layer marked as being in the computation state.

9. The method for optimizing model compression strategy of a large integrated model in an intelligent middleware platform according to claim 8, characterized in that, The method for performing lightweight compressed inference on the original large model includes: Perform data bypass operation on network layers marked as skipped; perform quantization tensor operation on network layers marked as computational state based on virtual quantization precision level to obtain compressed inference results; The percentage of skipped states in the statistical layer scheduling mask; the average entropy of the output probability distribution of all computational state network layers; the reciprocal of the weighted sum of the percentage and the average entropy is determined as the confidence index of the compressed inference result.

10. The method for optimizing model compression strategy of a large integrated model in an intelligent middleware platform according to claim 9, characterized in that, The method for quality verification of the confidence index includes: The confidence index is compared with the preset quality and safety threshold. If the confidence index is not less than the quality and safety threshold, the quality verification is deemed to have passed, and the compressed inference result is encapsulated and output. If the confidence index is less than the quality and safety threshold, the quality verification is deemed to have failed, and a full-precision re-inference instruction is triggered. Based on the full-precision re-inference instruction, the actual input data is input into the original large model to perform forward propagation, obtain the corrected compressed inference result, and encapsulate and output it. The actual input data and the corrected compressed inference results are stored as new samples in the task feature library.

Images