A method and system for multi-granularity dynamic pruning of generative AI models

Abstract

The application relates to the technical field of artificial intelligence model optimization, and discloses a multi-granularity dynamic pruning method and system for a generative AI model. The system comprises a model state acquisition module, which acquires intermediate layer output in real time through a probe, constructs a feature information set of neuron activation distribution, weight matrix norm and connection topology, and outputs a sparsification risk value based on the feature information set, an initial sparseness parameter and a real-time computing resource state; a pruning planning collaborative analysis and screening module generates a multi-granularity pruning scheme when the risk value exceeds a threshold value, calculates a collaborative interference value of an accuracy recovery operation and screens an optimal scheme; and a pruning strategy execution feedback module updates model parameters and generates a log feedback to a management terminal. The system realizes adaptive structure optimization and resource scheduling of a generative model in an inference process.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model optimization technology, specifically to a multi-granularity dynamic pruning method and system for generative AI models. Background Technology

[0002] Generative AI models, such as large-scale language models and multimodal generative models, face numerous challenges in practical deployment due to their massive number of parameters and high computational requirements. Especially in resource-constrained edge devices or high-concurrency service scenarios, model inference speed and computational cost become significant factors restricting their widespread application. Traditional model compression methods mainly include static pruning, quantization, and knowledge distillation. Static pruning is typically performed during model training, reducing model complexity by removing redundant weights or neurons. However, its fixed pruning strategy cannot adapt to the dynamically changing input features and computational environment during the inference phase. This lack of adaptability in compression methods easily leads to a sharp decline in model performance when dealing with out-of-distribution data.

[0003] Existing dynamic pruning methods are mostly focused on visual models or classification tasks, while research on dynamic pruning for generative models is still in its early stages. Generative tasks typically feature long output sequences, complex dependencies, and dynamically changing computation graphs. Directly applying existing dynamic pruning techniques can easily disrupt the coherence and semantic consistency of the generation process. Furthermore, most pruning methods do not fully consider the real-time state of underlying hardware resources, such as memory bandwidth, cache capacity, and processor load, resulting in the pruned model's speedup on actual hardware not meeting theoretical expectations.

[0004] Current generative model inference optimization often relies on pre-defined static configurations and cannot self-adjust based on real-time computational load and data characteristics. This optimization approach, lacking feedback and adaptive mechanisms, struggles to maintain a balance between stability and efficiency in diverse deployment environments. Therefore, a dynamic pruning system capable of real-time awareness of the model's internal state and external resource conditions, and performing multi-granular, recoverable pruning, is needed to support the efficient and robust deployment of generative AI models. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-granularity dynamic pruning method and system for generative AI models to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides a multi-granularity dynamic pruning system for generative AI models, the system comprising:

[0007] The model state acquisition module acquires intermediate layer outputs in real time by embedding probes into each layer of the generative AI model, and constructs a feature information set including neuron activation distribution, weight matrix norm, and connection topology.

[0008] The sparsity assessment module, based on the feature information set and combined with the initial sparsity parameters of the model, the current inference path, and the real-time computing resource status information, outputs the sparsity risk value of the model on the target hardware platform.

[0009] The pruning planning collaborative analysis and screening module generates multi-granularity pruning schemes based on the feature information set when the sparsity risk value exceeds the anomaly judgment threshold. It also combines the computing resource status information to calculate the collaborative interference value of the model accuracy recovery operation and screens the best dynamic pruning scheme.

[0010] The pruning strategy execution feedback module updates the model parameters according to the optimal dynamic pruning scheme, generates pruning logs, and feeds them back to the management terminal.

[0011] Preferably, the feature information set includes a hierarchical granularity partitioning unit, which decomposes the generative AI model into neuron-level, channel-level, and hierarchical pruning granularity levels.

[0012] The probes deployed in the model state acquisition module include activation value sampling probes, weight gradient probes, and computation delay probes.

[0013] Preferably, when the sparsity assessment module outputs a sparsity risk value, it performs overlap matching calculation between the current inference path and the norm of the weight matrix.

[0014] When the weight matrix norm distribution overlaps with the current inference path, the sparsification risk value is the product of the maximum weight norm of the overlapping region and the standard deviation of the activation distribution.

[0015] When the norm distribution of the weight matrix does not overlap with the current inference path, the sparsification risk value is the absolute value of the difference between the minimum weight norm and the standard deviation of the activation distribution.

[0016] Preferably, when the pruning planning collaborative analysis and screening module generates a multi-granularity pruning scheme, it obtains the pruning boundary feature nodes corresponding to each pruning granularity level in the feature information set.

[0017] In the current inference path, the topology of each pruning granularity level and its pruning boundary feature nodes are marked to generate a pruning planning region that does not destroy the topological connectivity of the model.

[0018] A heuristic search algorithm is used to generate candidate pruning schemes for each sub-region within the pruning planning area.

[0019] Preferably, the method for calculating the collaborative interference value by the pruning planning collaborative analysis screening module includes:

[0020] Obtain the number of retraining iterations required for model accuracy recovery, real-time computing resource utilization, and historical recovery operation time;

[0021] The collaborative interference value is quantified as a weighted harmonic average of the number of retraining iterations and the computational resource utilization rate;

[0022] When the historical recovery operation takes 0 minutes, the collaborative interference value is set to the preset maximum value.

[0023] Preferably, when the pruning planning collaborative analysis screening module calculates the interference impact value of each candidate pruning scheme on the model inference state:

[0024] The interference impact value of the k-th candidate pruning scheme is defined as the product of the sparsity risk value, the cooperative interference value, and the execution time of the scheme.

[0025] The execution time of the scheme is obtained by summing the product of the proportion of the processing time of each pruning level involved in the scheme to the total scheme time and the computational complexity of the level.

[0026] Preferably, when the pruning planning collaborative analysis and filtering module filters the optimal dynamic pruning scheme:

[0027] Select the candidate pruning scheme with the smallest interference value and its corresponding accuracy recovery operation identifier;

[0028] When the real-time computing resource status is lower than the resource threshold, the execution timing of the candidate pruning scheme is dynamically adjusted using a model predictive control algorithm.

[0029] Preferably, when the pruning strategy execution feedback module generates the pruning log:

[0030] The pruning log includes the pruning scheme's effective timestamp, the pruning boundary feature nodes used, accuracy recovery operation records, and the updated feature information set;

[0031] The consistency of the pruned model's output on the validation dataset and real-time input data is verified through a dual-channel verification mechanism.

[0032] Preferably, the pruning planning collaborative analysis and screening module includes:

[0033] The pruning scheme generation unit generates candidate pruning schemes based on pruning boundary feature nodes;

[0034] The precision interference analysis unit calculates the cooperative interference value and interference impact value of the precision recovery operation;

[0035] The dynamic pruning decision unit outputs the optimal dynamic pruning scheme based on resource constraints.

[0036] Preferably, the present invention also includes a multi-granularity dynamic pruning method for generative AI models, the method comprising all modules and method flows of the above-mentioned multi-granularity dynamic pruning system for generative AI models.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] This invention proposes a multi-granularity dynamic pruning system for generative AI models. By collecting real-time information on the model's internal state and external resources, the system achieves adaptive structural optimization during inference. The system can dynamically assess sparsity risk based on neuron activation distribution, weight matrix norm, and connection topology features, avoiding performance loss due to over-pruning. By introducing a multi-granularity pruning scheme generation mechanism, the system supports pruning operations at various granularities, from weights and neurons to attention heads, enhancing the flexibility and adaptability of the pruning strategy.

[0039] The system makes pruning decisions based on real-time computing resource status, ensuring that model compression matches hardware computing power and improving resource utilization efficiency. By restoring the cooperative interference value of computational accuracy recovery operations, the system can minimize the impact on model output quality during pruning, maintaining the semantic consistency and fluency of the generated tasks. The feedback mechanism after the pruning strategy is executed further optimizes the system's decisions, forming a closed-loop adjustment for continuous improvement.

[0040] This system is suitable for diverse deployment environments, performing particularly well in scenarios with fluctuating computing resources, enhancing the scalability and usability of generative AI models across edge devices and cloud services. The system achieves dynamic optimization without relying on pre-training or fixed configurations, reducing the need for manual parameter tuning and promoting the widespread deployment and application of efficient generative AI models. Attached Figure Description

[0041] Figure 1 This is a schematic diagram illustrating the working principle of the multi-granularity dynamic pruning method and system for generative AI models described in this invention.

[0042] Figure 2 Flowchart for calculating sparsity risk value;

[0043] Figure 3 A flowchart generated for a multi-granularity pruning scheme;

[0044] Figure 4 A flowchart for selecting the optimal dynamic pruning scheme;

[0045] Figure 5 This is a flowchart of the pruning planning collaborative analysis screening module. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Please see Figure 1 This invention provides a multi-granularity dynamic pruning method and system for generative AI models. The system includes: a model state acquisition module, a sparsity evaluation module, a pruning planning collaborative analysis and screening module, and a pruning strategy execution feedback module.

[0048] The model state acquisition module acquires intermediate layer outputs in real time through probes embedded in each layer of the generative AI model, constructing a feature information set including neuron activation distribution, weight matrix norm, and connection topology. Based on this feature information set, the sparsity assessment module, combined with the model's initial sparsity parameters, the current inference path, and real-time computing resource status information, outputs the sparsity risk value of the model on the target hardware platform. When the sparsity risk value exceeds the anomaly detection threshold, the pruning planning and collaborative analysis filtering module generates multi-granularity pruning schemes based on the feature information set, calculates the collaborative interference value of model accuracy recovery operations based on computing resource status information, and filters the optimal dynamic pruning scheme. The pruning strategy execution feedback module updates model parameters according to the optimal dynamic pruning scheme, generates pruning logs, and feeds them back to the management terminal. This system achieves efficient pruning of the model during inference through real-time monitoring and dynamic adjustment, balancing computing resources and model performance.

[0049] Example 1: See Figure 2The generative AI model is decomposed into different pruning granularity levels through hierarchical granularity partitioning units, including neuron-level, channel-level, and layer-level pruning. Neuron-level pruning involves the removal or retention of individual neurons, channel-level pruning involves the screening of the entire channel, and layer-level pruning considers the retention or removal of the entire network layer. This multi-granularity partitioning allows the system to make fine adjustments at different levels according to real-time needs, avoiding a one-size-fits-all pruning strategy that would lead to a sharp decline in model performance. The model state acquisition module deploys various probes to acquire the internal state of the model in real time. Among them, the activation value sampling probe is embedded in each layer to periodically collect the output values ​​of neurons, forming an activation distribution map. These sampled values ​​reflect the activity level of each part of the model when processing input data, and neurons with high activation values ​​usually contain more important information. The weight gradient probe monitors the changes in the weight matrix during training or inference and records the norm values ​​of the weights. These norms can represent the importance of the weights, and larger norms often correspond to more important connections. The computation delay probe measures the computation time of each layer. Combined with the inter-layer data transmission time, the connection topology of the model is constructed, showing the dependencies and computation paths between layers. All data collected by the probes are integrated into a feature information set, which is a dynamically updated data structure containing information such as timestamps, layer identifiers, and granularity types, providing comprehensive input for subsequent modules.

[0050] The sparsity assessment module calculates the sparsity risk value using a feature information set. First, it reads the initial sparsity parameters of the model, which are set during system initialization and define the acceptable basic sparsity level. The current inference path is extracted from the feature information set; it represents the sequence of neurons the model actually traverses when processing the current input. These sequences are composed of records from activation value sampling probes. Real-time computing resource status information is obtained from the system monitoring component, including data such as available memory, CPU load, and bandwidth. The core of the sparsity risk value calculation is overlap matching, comparing the current inference path with the norm of the weight matrix. The norm distribution of the weight matrix is ​​obtained from the feature information set, reflecting the importance distribution of each weight. If there is an overlap between the weight matrix norm distribution and the current inference path, it means that the currently activated path contains important connections with high weights. In this case, the sparsity risk value is calculated as the product of the maximum weight norm of the overlapping region and the standard deviation of the activation distribution. The maximum weight norm is selected from the overlapping region, and the standard deviation of the activation distribution is calculated based on the activation values ​​of neurons in the current layer, reflecting the dispersion of the activation values. This calculation method indicates that pruning may pose a significant risk when important paths are activated, thus resulting in a high risk value. If the norm distribution of the weight matrix does not overlap with the current inference path, it means that the currently activated path does not involve high-weight regions. In this case, the sparsity risk value is calculated as the absolute value of the difference between the minimum weight norm and the standard deviation of the activation distribution. The minimum weight norm is obtained from the entire weight matrix and represents the least important connection. The greater the difference between it and the standard deviation of the activation distribution, the greater the deviation of the current state from the ideal state, thus resulting in a higher risk value. The entire calculation process is performed in real time, relying on the continuous updating of the feature information set to ensure that the risk value reflects the latest state of the model.

[0051] In practice, the activation value sampling probe collects data at a fixed frequency, such as a full collection after processing a certain number of input samples, or triggered when a significant change in the activation distribution is detected. The weight gradient probe records data when weights are updated, or periodically extracts gradient information from the optimizer. The computational latency probe continuously monitors the execution time of each layer and constructs a topology graph by combining network latency data. The feature information set adopts a hierarchical storage structure for easy retrieval and updating. The risk calculation algorithm of the sparsity evaluation module is designed to be lightweight to avoid introducing excessive computational overhead and ensure that the system can respond in real time. Overlap matching calculation is implemented through efficient indexing and comparison algorithms to quickly locate overlapping areas or confirm non-overlapping states. The entire implementation emphasizes a balance between efficiency and accuracy, enabling the system to operate reliably in dynamic environments.

[0052] Example 2: See Figure 3The process begins with the pruning planning collaborative analysis and screening module parsing pruning boundary feature nodes corresponding to each pruning granularity level from the feature information set. These nodes are determined by analyzing the critical values ​​of neuron activation distribution and weight matrix norm. For example, at the neuron level, if the activation value of a neuron is consistently below a threshold and its connection weight norm is small, it may be marked as a boundary node. At the channel level, the average activation level and weight importance of the entire channel are comprehensively evaluated. At the layer granularity level, the computational contribution and resource consumption of the entire layer are taken into consideration. The identification of these boundary nodes relies on real-time updated data in the feature information set, ensuring the basic accuracy of pruning decisions.

[0053] In the current inference path, the system labels these identified pruning boundary feature nodes and their associated topologies. The labeling process needs to maintain the connectivity of the model's topology to avoid information flow interruptions caused by pruning. For example, in a multi-layer neural network, even if a layer is labeled as a candidate pruning layer, its connections with preceding and following layers need to be checked to ensure that removing that layer will not cause irreparable structural damage. After labeling, the system generates one or more pruning planning regions. These regions are structurally independent subnetworks enclosed by boundary nodes, and performing pruning operations within these regions will not affect the overall topological integrity of the model.

[0054] When generating candidate pruning schemes, the system employs a heuristic search algorithm to explore the pruning planning region. This algorithm avoids getting trapped in local optima. During the search process, the algorithm evaluates the potential impact of different pruning combinations on model performance, generating multiple candidate schemes. Each scheme specifies in detail the specific pruning operations and their execution order at the neuron, channel, or layer level.

[0055] The calculation of the co-interference value is another core step. The system first obtains the number of retraining iterations required for model accuracy recovery, data sourced from historical operation records or predictions based on the current model state. Real-time computing resource utilization is obtained through the system monitoring interface, reflecting the current availability of computing resources. Historical recovery operation time is extracted from the log database, providing a reference for current evaluation. The co-interference value is quantified using a weighted harmonic mean, incorporating the number of retraining iterations and computing resource utilization into a unified calculation framework. The weighting coefficients are configured according to specific application scenarios; for example, in resource-constrained environments, the weight of resource utilization may be higher. This calculation method balances the iterative requirements of the recovery process with resource consumption. When the system detects that historical recovery operation time records are empty, it sets the co-interference value to a higher preset value, reflecting the uncertainty caused by the lack of historical data.

[0056] The entire implementation process emphasizes the coordination and dynamic adjustment of each stage. The generation of pruning schemes is not static but continuously optimized as the feature information set is updated. The calculation of collaborative interference values ​​is also not a one-time event but is continuously performed during the pruning planning process to ensure that the final selected scheme meets the sparsity requirements while minimizing interference with model accuracy. Through this refined implementation method, the system achieves intelligent pruning decisions in dynamic environments.

[0057] Example 3: See Figure 4 This focuses on the quantification of the impact of disturbances and the selection and decision-making process for the optimal dynamic pruning scheme. The core of this implementation process lies in establishing a comprehensive evaluation framework that incorporates sparsity risk, recovery operation disturbances, and execution efficiency into a unified consideration system.

[0058] The interference impact value is calculated using the following formula:

[0059]

[0060] in: Indicates the first The interference impact value of each candidate pruning scheme, This represents the sparsity risk value output from the sparsity assessment module. This represents the cooperative interference value for accuracy recovery operations. This is the execution time of the candidate pruning scheme. This product relationship reflects the system's comprehensive trade-offs over multiple factors; a higher sparsity risk value, a larger cooperative interference, or a longer execution time will all lead to an increase in the interference impact value.

[0061] Solution execution time The computation requires a deep dive into the specific operations at each pruning granularity level. The system first identifies all pruning granularity levels involved in the scheme, including the neuron level, channel level, and the specific operation objects within each level. For each level, its processing time data is extracted, derived from computational latency probe records in the feature information set. Simultaneously, the system assigns a computational complexity coefficient to each level, predefined based on the operation type and structural characteristics of that level. For example, neuron-level pruning has relatively low computational complexity, while operations involving the entire channel or level have higher complexity. Time consumption is calculated by accumulating the product of the proportion of processing time at each level to the total time and the corresponding computational complexity. This weighted accumulation method more accurately reflects the contribution of different level operations to the overall time consumption.

[0062] After obtaining the interference impact values ​​of all candidate pruning schemes, the system enters the screening and decision-making stage. The screening principle is based on minimum value selection, that is, selecting the candidate pruning scheme with the smallest interference impact value as the optimal dynamic pruning scheme. At the same time, the system records the accuracy recovery operation identifier corresponding to the scheme. This identifier indicates the type and intensity of accuracy recovery measures to be taken after executing the pruning scheme.

[0063] The decision-making process also needs to consider the constraints of real-time computing resource status. When the real-time computing resource status is detected to be lower than a preset resource threshold, the system initiates a model predictive control algorithm for dynamic adjustment. This algorithm constructs a predictive model to predict resource usage and the effectiveness of pruning schemes over a future period. Based on these predictions, the algorithm re-plans the execution sequence of candidate pruning schemes, potentially adopting strategies such as delaying the execution of certain high-resource-consuming schemes and prioritizing the execution of low-interference schemes. During the timing adjustment process, the system continuously monitors changes in resource status and dynamically updates the predictive model, forming a closed-loop control mechanism.

[0064] The entire implementation process demonstrates the system's adaptive characteristics. The calculation of interference impact values ​​is not static but dynamically adjusted as the feature information set is updated and resource status changes. The screening decision mechanism is also flexible, intelligently adjusting the scheme execution strategy under resource constraints. This implementation method ensures that the system can make reasonable pruning decisions under various operating environments, satisfying the model sparsity requirements while minimizing interference with the normal inference process.

[0065] The implementation of model predictive control algorithms requires establishing a spatial representation of the system state, including parameters such as current resource utilization, queue of pending execution plans, and historical execution results. The algorithm employs a rolling optimization strategy, recalculating the optimal execution sequence within each control cycle. This mechanism enables the system to respond promptly to environmental changes, maintaining the timeliness and accuracy of decision-making.

[0066] Example 4: This involves a mechanism for generating and verifying pruning logs. The implementation process begins with the data recording phase after the pruning strategy execution feedback module completes the pruning operation. The system first obtains the timestamp of the pruning scheme's effectiveness. This timestamp accurately records the actual moment the pruning operation takes effect on the model, typically using a high-precision clock source and synchronized to all nodes in the distributed system. The timestamp's recording format includes year, month, day, hour, minute, second, and millisecond, as well as time zone information, ensuring log time consistency in a distributed environment.

[0067] The system records the pruning boundary feature node information. This node information details the specific identifiers of neurons, channels, or layers that were removed or retained during the actual pruning operation. Each node's record includes its layer location, node type, original parameter values, and state changes after pruning. For example, in pruning a visual Transformer model, it might record a specific dimension removed from an attention head, or a set of neurons pruned in a multilayer perceptron.

[0068] The accuracy recovery operation log is a crucial component of the log. This section details any accuracy recovery measures performed after pruning, including the number of retraining iterations, the optimization algorithm parameters used, the learning rate adjustment strategy, and the trend of loss function changes during the recovery process. If special recovery techniques such as knowledge distillation were employed, the specific teacher-student network structure and distillation temperature parameters will also be recorded here. The updated feature information set is the core data block in the log. This set reflects the latest state of the model after pruning, including the modified neuron activation distribution, weight matrix norm distribution, and adjusted connection topology. This data is stored in a structured format for easy subsequent query analysis and model version comparison; see Table 1.

[0069] Table 1: Pruning log data structure.

[0070]

[0071] The dual-channel validation mechanism is implemented using two parallel channels. The first channel runs the original, unpruned version of the model, while the second channel runs the newly pruned model. Both channels receive the same input data, including samples extracted from the validation dataset and a real-time input data stream. The validation dataset typically contains a representative set of samples covering various typical use cases and boundary conditions of the model.

[0072] The validation process compares the outputs of two channels, including not only the final prediction but also the feature representations of intermediate layers. The system calculates similarity metrics between output vectors, such as cosine similarity or Euclidean distance, and sets reasonable difference thresholds. When the output difference exceeds the predetermined threshold, the system records detailed difference information, including the specific level of the difference, its degree, and the corresponding input data features. During consistency validation, the system employs a hierarchical validation strategy. It first compares the overall results of the output layer; if significant differences are found, it delves deeper into the intermediate layers for layer-by-layer analysis. This hierarchical validation method helps to quickly pinpoint the specific location of the problem. Validation results are logged in real time, including the number of samples that passed validation, detailed analysis reports of failed cases, and overall consistency metrics.

[0073] The entire implementation process emphasizes data integrity and traceability. Pruning logs employ encrypted storage and access control mechanisms to ensure data confidentiality and integrity. The log management system provides multi-version management capabilities, allowing for the retrospective analysis of any historical pruning operation, supporting model evolution analysis and performance auditing. The dual-channel verification mechanism prioritizes operational efficiency, employing asynchronous parallel processing to minimize impact on the normal inference process.

[0074] Log data is stored in a columnar format for easy and rapid querying and analysis. The system has established a robust log indexing mechanism, supporting fast retrieval based on various criteria such as time range, pruning type, and model level. A regular log archiving strategy ensures the manageability of long-term data storage while maintaining fast access to recent data.

[0075] Example 5: See Figure 5 This document describes the internal unit structure and functional implementation of a pruning planning collaborative analysis and screening module. This module comprises three core units: a pruning scheme generation unit, a precision interference analysis unit, and a dynamic pruning decision unit. These units work collaboratively to complete the entire process from scheme generation to final decision. The main function of the pruning scheme generation unit is to generate candidate pruning schemes based on pruning boundary feature nodes. This unit first obtains pre-identified pruning boundary feature nodes from a feature information set. These nodes are distributed at different granularity levels, such as neuron level, channel level, and layer level. The acquisition process includes reading the node's location information, type identifier, and its connection relationship in the model topology. Based on this node information, the unit uses a heuristic search algorithm to generate multiple candidate pruning schemes. Each scheme specifies the detailed pruning operations, including the set of neurons to be removed, the list of channels to be retained, and the network layers that may be completely removed. During scheme generation, the algorithm considers the correlation between nodes to avoid generating pruning schemes that compromise the integrity of the model's functionality. All generated candidate schemes are accompanied by metadata information, including scheme number, generation timestamp, and the granularity level involved.

[0076] The accuracy interference analysis unit is responsible for calculating the cooperative interference value of the accuracy recovery operation and the interference impact value of each scheme. This unit first obtains the relevant parameters required for the model accuracy recovery operation, including the number of retraining iterations, real-time computing resource utilization, and historical recovery operation time data. This data is extracted from the system monitoring component and historical record database. The cooperative interference value is calculated using a weighted harmonic average method, incorporating the number of retraining iterations and computing resource utilization into a unified calculation framework. The weighting coefficients are set considering the needs of specific application scenarios, and different weight configurations may be used in different operating environments. The interference impact value is calculated by combining the sparsity risk value obtained from the sparsity evaluation module, the calculated cooperative interference value, and the scheme execution time. During the calculation process, the unit considers the dynamic changes of each factor to ensure the timeliness of the calculation results.

[0077] The dynamic pruning decision unit outputs the optimal dynamic pruning scheme based on real-time resource constraints. This unit receives all candidate pruning schemes and their corresponding interference impact values. The decision-making process is based on comparative analysis, selecting the candidate scheme with the smallest interference impact value as the optimal scheme. Simultaneously, the unit records the accuracy recovery operation identifier corresponding to this scheme, indicating the type of accuracy recovery measure to be taken. During the decision-making process, the unit continuously monitors the real-time computing resource status, including memory usage, processor load, network bandwidth, and other indicators. When the resource status is detected to be below a preset threshold, the unit activates a model predictive control algorithm to dynamically adjust the execution timing of candidate schemes. The adjustment process is based on the predictive model's estimation of future resource usage, and may adopt strategies such as delaying the execution of high-resource-consuming schemes and prioritizing the execution of low-interference schemes. The final output of the optimal dynamic pruning scheme includes complete execution instructions and resource allocation suggestions.

[0078] An effective data exchange mechanism is established between these three units. The pruning scheme generation unit transmits the generated candidate schemes to the precision interference analysis unit for analysis and calculation. The precision interference analysis unit then transmits the calculation results to the dynamic pruning decision unit for final decision-making. The entire data flow process adopts an asynchronous processing method, allowing each unit to work in parallel and improving the overall system efficiency. Communication between units uses a standardized data format to ensure the accuracy and consistency of information transmission. The system also establishes an error handling mechanism; when one unit malfunctions, other units can adjust their working status accordingly to ensure the stable operation of the system.

[0079] The generation speed and accuracy of the pruning scheme generation unit are balanced with the processing capacity of the interference analysis unit to avoid scheme backlog. The decision frequency of the dynamic pruning decision unit is dynamically adjusted according to the system's operating status, increasing the decision frequency when resources are scarce and appropriately decreasing the frequency when resources are abundant to save computational overhead. The operating status of the entire module is continuously monitored, including indicators such as processing latency and resource consumption of each unit, to ensure that the system is always in optimal working condition.

[0080] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0081] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-granularity dynamic pruning system for generative AI models, characterized in that, include: The model state acquisition module acquires intermediate layer outputs in real time by embedding probes into each layer of the generative AI model, and constructs a feature information set including neuron activation distribution, weight matrix norm, and connection topology. The sparsity assessment module, based on the feature information set and combined with the initial sparsity parameters of the model, the current inference path, and the real-time computing resource status information, outputs the sparsity risk value of the model on the target hardware platform. The pruning planning collaborative analysis and screening module generates multi-granularity pruning schemes based on the feature information set when the sparsity risk value exceeds the anomaly judgment threshold. It also combines the computing resource status information to calculate the collaborative interference value of the model accuracy recovery operation and screens the best dynamic pruning scheme. The pruning strategy execution feedback module updates the model parameters according to the optimal dynamic pruning scheme, generates pruning logs, and feeds them back to the management terminal.

2. The multi-granularity dynamic pruning system for generative AI models according to claim 1, characterized in that, The feature information set includes a hierarchical granularity partitioning unit, which decomposes the generative AI model into neuron-level, channel-level, and hierarchical pruning granularity levels. The probes deployed in the model state acquisition module include activation value sampling probes, weight gradient probes, and computation delay probes.

3. The multi-granularity dynamic pruning system for generative AI models according to claim 2, characterized in that, When the sparsity assessment module outputs a sparsity risk value, it performs overlap matching calculation between the current inference path and the norm of the weight matrix. When the weight matrix norm distribution overlaps with the current inference path, the sparsification risk value is the product of the maximum weight norm of the overlapping region and the standard deviation of the activation distribution. When the norm distribution of the weight matrix does not overlap with the current inference path, the sparsification risk value is the absolute value of the difference between the minimum weight norm and the standard deviation of the activation distribution.

4. The multi-granularity dynamic pruning system for generative AI models according to claim 2, characterized in that, When the pruning planning collaborative analysis and screening module generates a multi-granularity pruning scheme, it obtains the pruning boundary feature nodes corresponding to each pruning granularity level in the feature information set. In the current inference path, the topology of each pruning granularity level and its pruning boundary feature nodes are marked to generate a pruning planning region that does not destroy the topological connectivity of the model. A heuristic search algorithm is used to generate candidate pruning schemes for each sub-region within the pruning planning area.

5. The multi-granularity dynamic pruning system for generative AI models according to claim 4, characterized in that, The method for calculating the collaborative interference value in the pruning planning collaborative analysis and screening module includes: Obtain the number of retraining iterations required for model accuracy recovery, real-time computing resource utilization, and historical recovery operation time; The collaborative interference value is quantified as a weighted harmonic average of the number of retraining iterations and the computational resource utilization rate; When the historical recovery operation takes 0 minutes, the collaborative interference value is set to the preset maximum value.

6. The multi-granularity dynamic pruning system for generative AI models according to claim 5, characterized in that, When the pruning planning collaborative analysis and screening module calculates the interference impact value of each candidate pruning scheme on the model inference state: The interference impact value of the k-th candidate pruning scheme is defined as the product of the sparsity risk value, the cooperative interference value, and the execution time of the scheme. The execution time of the scheme is obtained by summing the product of the proportion of the processing time of each pruning level involved in the scheme to the total scheme time and the computational complexity of the level.

7. The multi-granularity dynamic pruning system for generative AI models according to claim 1, characterized in that, When the pruning planning collaborative analysis and filtering module filters the optimal dynamic pruning scheme: Select the candidate pruning scheme with the smallest interference value and its corresponding accuracy recovery operation identifier; When the real-time computing resource status is lower than the resource threshold, the execution timing of the candidate pruning scheme is dynamically adjusted using a model predictive control algorithm.

8. The multi-granularity dynamic pruning system for generative AI models according to claim 1, characterized in that, When the pruning strategy execution feedback module generates the pruning log: The pruning log includes the pruning scheme's effective timestamp, the pruning boundary feature nodes used, accuracy recovery operation records, and the updated feature information set; The consistency of the pruned model's output on the validation dataset and real-time input data is verified through a dual-channel verification mechanism.

9. The multi-granularity dynamic pruning system for generative AI models according to claim 1, characterized in that, The pruning planning collaborative analysis and filtering module includes: The pruning scheme generation unit generates candidate pruning schemes based on pruning boundary feature nodes; The precision interference analysis unit calculates the cooperative interference value and interference impact value of the precision recovery operation; The dynamic pruning decision unit outputs the optimal dynamic pruning scheme based on resource constraints.

10. A multi-granularity dynamic pruning method for generative AI models, characterized in that, The system includes all modules and method flows of the multi-granularity dynamic pruning system for generative AI models as described in any one of claims 1 to 9.

Images