Dynamic pruning method and device for edge-oriented hybrid expert model calculation

Abstract

This invention proposes a dynamic pruning method for edge-side hybrid expert model computation, comprising: performing forward computation of the gating network during inference, bypassing the acquisition of expert indices and corresponding routing weights output by each layer of the gating network, and statistically analyzing the activation state distribution characteristics of each layer of expert modules; calculating a comprehensive importance score vector based on the activation state distribution characteristics of each layer of expert modules to adaptively select the expert sub-networks to be retained for each layer of expert modules; reconstructing a lightweight hybrid expert model based on the expert sub-networks to be retained for each layer of expert modules; and releasing the GPU memory and main memory resources occupied by the full hybrid expert model during the idle window of the inference service, completing the online switching between the lightweight hybrid expert model and the full hybrid expert model. This method achieves dynamic release of redundant expert resources, thereby reducing storage and memory access overhead during edge inference and improving overall inference efficiency and resource utilization.

Description

Technical Field

[0001] This invention relates to the field of edge devices, and more particularly to a dynamic pruning method and apparatus for edge-side hybrid expert model computation. Background Technology

[0002] Hybrid expert (MoE) models activate only a small number of expert subnetworks during the inference phase through gating networks, thereby expanding the model parameter scale while maintaining a relatively controllable single computational cost. This type of model can improve model capacity and expressive power without significantly increasing computational complexity, thus achieving good results in scenarios such as natural language processing, code generation, and multimodal tasks.

[0003] However, the overall parameter size of hybrid expert models, which consist of numerous expert sub-networks, has not decreased proportionally. Edge computing devices typically still need to store the complete set of expert parameters in local flash memory, RAM, or video memory during deployment and operation. Therefore, the overall storage footprint, memory access overhead, and cross-device data transfer pressure caused by the model remain significant.

[0004] Especially in low-resource, heterogeneous hardware environments, the system is constrained by hardware resources such as physical storage space, CPU main memory capacity, GPU video memory, and bus bandwidth. Existing solutions mainly employ static parameter loading or offline pruning strategies. This involves pruning or fixing the expert set based on global statistics or empirical rules before model deployment and maintaining this structure unchanged during inference service operation. These methods offer advantages in terms of implementation complexity and system stability, and can reduce model resource consumption to some extent. However, due to their lack of dynamic awareness of task characteristics and actual expert usage during inference, resource allocation remains based on global or average distribution assumptions, making it difficult to achieve fine-grained adaptation to different task scenarios and fully adapt to the resource-constrained characteristics of edge hardware. Particularly in semantically focused application scenarios such as continuous dialogue, local code generation, and mathematical reasoning, expert invocation in the MoE model typically exhibits a significant imbalance, with a few frequently activated experts while a large number of low-frequency experts continue to occupy edge storage, memory, and video memory resources, resulting in low hardware resource utilization. Summary of the Invention

[0005] This invention provides a dynamic pruning method, apparatus, and storage medium for edge-side hybrid expert model computation. The method enables the dynamic release of redundant expert resources, thereby reducing storage and memory access overhead during edge inference and improving overall inference efficiency and resource utilization.

[0006] This invention provides a dynamic pruning method for edge-end hybrid expert model computation, comprising:

[0007] The full hybrid expert model is loaded into the video memory of the edge computing acceleration unit at the edge control unit and the inference service is started. The hybrid expert model includes multi-layer expert modules, and each layer of expert modules includes multiple expert sub-networks and gating networks.

[0008] During inference, the edge computing acceleration unit performs forward computation of the gating network, bypasses the collection of expert indexes and corresponding routing weights output by each layer of the gating network, and statistically analyzes the activation state distribution characteristics of each layer of expert modules.

[0009] Based on the activation state distribution characteristics of each expert module, a comprehensive importance score vector is calculated to adaptively select the expert subnetworks to be retained for each expert module.

[0010] The edge-side control unit reconstructs a lightweight hybrid expert model based on the expert sub-networks to be retained from each layer of expert modules;

[0011] During the idle window of the inference service, the edge-side control unit releases the video memory and main memory resources occupied by the full hybrid expert model, and loads the lightweight hybrid expert model to the edge-side computing acceleration unit, completing the online switching between the lightweight hybrid expert model and the full hybrid expert model.

[0012] In one embodiment of the present invention, the steps of bypassing the acquisition of expert indexes and corresponding routing weights output by each layer of the gating network, statistically analyzing the activation state distribution characteristics of each layer of expert modules, and storing them in the video memory of the edge computing device include:

[0013] A bypass activation tracing operator is deployed at the output of each layer of the gated network. The tracing operator is decoupled from the main inference path bypass and obtains the expert index and routing weight of the gated network output in read-only mode without modifying the tensor content and calculation results of the original computation graph.

[0014] During the inference process, for each input token, the gating network selects Top-k experts to perform feedforward computation, and the tracking operator calls the atomic update kernel of the edge computing acceleration unit, concurrently calculates the activation frequency value and cumulative routing weight value of each expert sub-network, and stores the statistical results in the global memory of the edge computing acceleration unit.

[0015] The activation state distribution feature includes activation frequency value and cumulative routing weight value.

[0016] In one embodiment of the present invention, the concurrent statistics process ends when the cumulative number of processed tokens reaches a preset trigger threshold.

[0017] In one embodiment of the present invention, a comprehensive importance score vector is calculated based on the activation state distribution characteristics of each layer of expert modules, including:

[0018] For the hybrid expert model, the first The layer expert module reads the activation frequency vector and routing weight vector of the layer, calculates the normalized distribution and distribution sharpness of the activation frequency vector, and calculates the normalized distribution and distribution sharpness of the routing weight vector; wherein, each element in the activation frequency vector and routing weight vector corresponds to an expert subnetwork, and records the cumulative activation count and routing weight value of the expert subnetwork respectively.

[0019] The adaptive balance coefficient is calculated based on the distribution sharpness of the activation frequency vector and the distribution sharpness of the routing weight vector.

[0020] Based on the normalized distribution of the activation frequency vector, the normalized distribution of the routing weight vector, and the adaptive balancing coefficient, the comprehensive importance score vector is calculated, expressed as:

[0021]

[0022] in, This is a comprehensive importance score vector, where each element corresponds to the importance score of a single expert subnetwork. For adaptive balance coefficients, Activate the frequency vector for experts; This is the routing weight vector.

[0023] In one embodiment of the present invention, adaptively selecting the expert sub-networks to be retained for each layer of expert modules includes:

[0024] The first The elements in the comprehensive importance score vector of the layer expert module are sorted in descending order, and the current element is added to all previous elements in turn to calculate the cumulative score and the proportion of the total score.

[0025] When the cumulative score and its proportion of the total score reach the preset coverage threshold, the expert subnetworks corresponding to the elements before the current element are determined as expert subnetworks to be retained, and the expert subnetworks corresponding to the remaining elements are determined as expert subnetworks to be pruned.

[0026] In one embodiment of the present invention, reconstructing a lightweight hybrid expert model based on the expert sub-networks to be retained from each layer of expert modules includes:

[0027] Create a new model file, and only write the weight tensors corresponding to the expert subnetworks to be retained in each layer into the new model file. Do not write the weight tensors of the experts to be pruned into the new model file.

[0028] The indexes of the expert subnetworks to be retained in each layer are renumbered sequentially to obtain the lightweight hybrid expert model.

[0029] In one embodiment of the present invention, the request status of the inference service is detected in real time. When no active inference request is detected, a termination signal is sent to the original inference service process. After the original inference service process exits, the video memory and main memory resources occupied by the full hybrid expert model are released.

[0030] In one embodiment of the present invention, the model file loading pointer of the inference service is redirected to the lightweight hybrid expert model, a new inference service process is started, and the online model switching is completed.

[0031] Another aspect of the present invention provides a dynamic pruning device based on hybrid expert model calculations, comprising:

[0032] The loading module is used to load the full hybrid expert model and start the inference service. The hybrid expert model contains multiple expert modules, and each expert module contains multiple expert sub-networks and gating networks.

[0033] The bypass acquisition module is used to perform forward computation of the gating network during inference, bypass acquisition of expert indexes and corresponding routing weights output by each layer of the gating network, and statistical analysis of the activation state distribution characteristics of each layer of expert modules.

[0034] An adaptive filtering module is used to calculate a comprehensive importance score vector based on the activation state distribution characteristics of each layer of expert modules, so as to adaptively filter out the expert subnetworks to be retained for each layer of expert modules.

[0035] The model reconstruction module is used to reconstruct a lightweight hybrid expert model based on the expert subnetworks to be retained from each layer of expert modules.

[0036] The model switching module is used to release the resources occupied by the full hybrid expert model during the idle window of the inference service and load the lightweight hybrid expert model to complete the online switching between the lightweight hybrid expert model and the full hybrid expert model.

[0037] In another aspect, the present invention provides a readable storage medium, characterized in that the readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the above-described dynamic pruning method for edge-oriented hybrid expert model computation.

[0038] As can be seen from the above solutions, the advantages of the present invention are:

[0039] This invention provides a dynamic pruning method for edge-side hybrid expert model computation. During inference, it performs forward computation of the gating network, bypassing the acquisition of expert indices and corresponding routing weights output by each layer of the gating network, and statistically analyzes the activation state distribution characteristics of each layer's expert modules. Based on these characteristics, a comprehensive importance score vector is calculated to adaptively select the expert sub-networks to be retained for each layer. A lightweight hybrid expert model is then reconstructed based on these retained expert sub-networks. Finally, the GPU and main memory resources occupied by the full hybrid expert model are released, enabling online switching between the lightweight and full hybrid expert models. This method transforms the expert activation distribution characteristics generated by the hybrid expert model during edge-side task inference into a basis for scheduling edge-side hardware resources. This further extends the sparse computation characteristics at the model structure level into a hardware resource optimization mechanism at the edge deployment and runtime levels, providing a feasible path for large-scale model deployment in edge-side hardware-constrained environments. Specifically, by physically stripping low-frequency redundant expert subnetworks from each layer of expert modules, a lightweight hybrid expert model is generated. This model effectively reduces the overall storage size and runtime memory usage of the edge-side model while ensuring the basic stability of the model's core inference capabilities. It also improves end-to-end inference efficiency and resource utilization, adapts to the hardware characteristics of edge devices with limited storage capacity, and enables efficient inference deployment for resource-constrained environments. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall process of a dynamic pruning method for edge-end hybrid expert model computation provided in an embodiment of the present invention;

[0041] Figure 2 for Figure 1 A detailed flowchart of step S2 is shown below;

[0042] Figure 3 for Figure 1 A detailed flowchart of step S3 is shown below;

[0043] Figure 4 This is a general block diagram of a dynamic trimming device calculated using a hybrid expert model, provided for another embodiment of the present invention.

[0044] The attached figures are labeled as follows:

[0045] 400: Dynamic clipping device calculated using a hybrid expert model;

[0046] 410: Load module;

[0047] 420: Bypass acquisition module;

[0048] 430: Adaptive filtering module;

[0049] 440: Model Reconstruction Module;

[0050] 450: Model switching module. Detailed Implementation

[0051] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0052] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0053] As mentioned in the background section, research on the deployment of hybrid expert models on edge devices revealed that although the MoE model activates only a small number of experts during the inference phase through a gating network, in actual deployment, physical weight space is usually reserved for all experts. This results in a significant mismatch between the model's resource consumption and the actual number of experts participating in the computation. Particularly in semantically focused scenarios such as continuous dialogue, mathematical reasoning, and code generation, some experts consistently become the primary routing targets, while others remain in a low-frequency activation state for extended periods. In other words, there is a structural mismatch between actual resource consumption and the true contributions of experts, which makes dynamic expert pruning based on task characteristics possible. To address this, this application considers that if the activation status of experts can be perceived in real-time during operation, and the expert set can be dynamically adjusted accordingly, it may be possible to effectively release the resources occupied by low-frequency experts without significantly affecting the model's inference capabilities, thereby effectively utilizing the hardware resources of edge devices.

[0054] Specifically, in one embodiment, reference Figure 1 As shown, Figure 1 A schematic diagram of the overall process of a dynamic pruning method for edge-side hybrid expert model computation provided in one embodiment is shown.

[0055] A dynamic pruning method for edge-end hybrid expert model computation includes the following steps:

[0056] Step S1: Load the full hybrid expert model into the video memory of the edge computing acceleration unit in the edge-side control unit and start the inference service. The hybrid expert model includes multi-layer expert modules, and each layer of expert modules includes multiple expert sub-networks and gating networks.

[0057] In one embodiment, in a heterogeneous edge computing device, edge-side control units (e.g., CPUs) and edge-side computing acceleration units (e.g., GPUs) together constitute the edge-side computing resources, and the two are connected via a high-speed bus. Model inference is mainly performed on the GPU side, while the CPU side is responsible for background model reconstruction and resource management.

[0058] The full hybrid expert model is loaded into the video memory of the edge computing acceleration unit at the edge control unit and the inference service is started. In the initial inference process, inference is performed according to the full hybrid expert model structure.

[0059] Step S2: During the inference process, the edge computing acceleration unit performs forward computation of the gating network, bypasses the collection of expert indexes and corresponding routing weights output by each layer of the gating network, and statistically analyzes the activation state distribution characteristics of each layer of expert modules.

[0060] In one embodiment, an activation tracking operator is introduced into the computation graph of the inference engine to bypass the gated network output and obtain expert indexes and corresponding routing weight information without affecting the main computation path, thereby achieving low-overhead statistics on expert activation states. Specifically, refer to... Figure 2 As shown, Figure 2 A schematic diagram of the specific process for step S2 is shown.

[0061] The steps include: bypassing the acquisition of expert indexes and corresponding routing weights output by each layer of the gating network, statistically analyzing the activation state distribution characteristics of each layer of expert modules, and storing them in the video memory of the edge computing device.

[0062] Step S21: Deploy a bypass activation tracing operator at the output of each layer of the gated network. The tracing operator is decoupled from the main inference path bypass, and the tracing operator obtains the expert index and routing weight of the gated network output in read-only mode without modifying the tensor content and calculation results of the original computation graph.

[0063] Step S22: During the inference process, for each input Token, the gating network selects Top-k experts to perform feedforward computation, and the tracking operator calls the atomic update kernel of the edge-side computing acceleration unit, concurrently calculating the activation frequency value and cumulative routing weight value of each expert sub-network, and storing the statistical results in the global memory of the edge-side computing acceleration unit. The activation state distribution characteristics include the activation frequency value and the cumulative routing weight value.

[0064] In this embodiment, a non-intrusive activation tracking operator is introduced at the output position of the gating network. Expert indexes and corresponding routing weight information are collected in a bypass manner, and a concurrent statistics mechanism is used to summarize and analyze the activation frequency and cumulative routing weight values ​​of each expert sub-network. This enables real-time and refined perception of the expert routing distribution corresponding to the current task context during model operation without interfering with the main inference calculation path. It continuously acquires activation frequency and cumulative routing weight values ​​reflecting the characteristics of the current task, providing a reliable data foundation for subsequent dynamic pruning, while keeping additional computation and system overhead at a low level.

[0065] Furthermore, in one embodiment, when the cumulative number of processed tokens reaches a preset trigger threshold, the concurrent statistics process ends, and the process proceeds to the dynamic pruning decision process in step S3. This preset trigger threshold is used to avoid premature pruning decisions when the statistical sample is insufficient. In one specific implementation, the preset trigger threshold is set to one million tokens.

[0066] Step S3: Calculate the comprehensive importance score vector based on the activation state distribution characteristics of each layer of expert modules, so as to adaptively select the expert subnetworks to be retained for each layer of expert modules.

[0067] In one embodiment, a comprehensive importance score vector is calculated based on the activation state distribution characteristics of each layer of expert modules. The pruning strategy is dynamically adjusted, and expert subnetworks to be retained are adaptively selected for each layer of expert modules, thus determining a reasonable set of expert subnetworks to be retained. Specifically, refer to... Figure 3 As shown, Figure 3 A schematic diagram of the specific process for step S3 is shown.

[0068] Based on the activation state distribution characteristics of each expert module, the comprehensive importance score vector is calculated to adaptively select the expert subnetworks to be retained for each expert module. The specific steps include:

[0069] Step S31, for the hybrid expert model... The layer expert module reads the activation frequency vector and routing weight vector of that layer, calculates the normalized distribution and distribution sharpness of the activation frequency vector, and calculates the normalized distribution and distribution sharpness of the routing weight vector. Each element in the activation frequency vector and routing weight vector corresponds to an expert subnetwork, recording the cumulative activation count and routing weight value of that expert subnetwork, respectively. The distribution sharpness is used to characterize the... The sharpness of the concentration of activation frequency / routing weight in the layer expert module indicates the degree of concentration. Higher sharpness means that a few top experts occupy the vast majority of activations or routing weights, resulting in higher redundancy of long-tail experts. Lower sharpness indicates a more balanced use of experts, requiring more experts to be retained to maintain model performance. In a specific implementation, the Gini coefficient is used to calculate the distribution sharpness, and the calculation method is as follows: , among which is the normalized value of the activation frequency / routing weight of the i-th expert subnetwork, and N is the total number of expert subnetworks in this layer; The closer it is to 1, the sharper the distribution and the higher the concentration. The closer it is to 0, the more uniform the distribution.

[0070] Step S32: Calculate the adaptive balance coefficient based on the distribution sharpness of the activation frequency vector and the distribution sharpness of the routing weight vector.

[0071] Step S33: Based on the normalized distribution of the activation frequency vector, the normalized distribution of the routing weight vector, and the adaptive balancing coefficient, calculate the comprehensive importance score vector, expressed as:

[0072] in, This is a comprehensive importance score vector, where each element corresponds to the importance score of a single expert subnetwork. For adaptive balance coefficients, Activate the frequency vector for experts; This is the routing weight vector.

[0073] Step S34, the first The elements in the comprehensive importance score vector of the layer expert module are sorted in descending order, and the current element is summed with all previous elements to calculate the cumulative score and its proportion to the total score. Specifically, the elements in the vector are sorted in descending order, and the current element is summed with all previous elements to calculate the cumulative score and its proportion to the total score. After all expert subnetworks in the layer expert module are sorted in descending order of importance score, starting from the first-ranked element, the importance scores of the current element and all preceding elements are accumulated sequentially to obtain the cumulative score. The total score is the [number]th [level]. The sum of importance scores for all expert subnetworks in the layer expert module.

[0074] Step S35: When the cumulative score and its proportion to the total score reach a preset coverage threshold, the expert subnetworks corresponding to the elements preceding the current element are identified as expert subnetworks to be retained, and the expert subnetworks corresponding to the remaining elements are identified as expert subnetworks to be pruned. In a specific implementation, the coverage threshold can be set according to the model accuracy requirements and resource release goals, for example, it can be set to 80% to 95%.

[0075] In this embodiment, to address the differences in expert activation distribution across different levels, a comprehensive importance assessment mechanism combining activation frequency and routing weights is constructed. The weights of various indicators are dynamically adjusted based on distribution characteristics, and a cumulative coverage strategy is used to determine the set of expert subnetworks to be retained at each level. Based on the expert usage pattern driven by task semantics, adaptive filtering and retention of core expert subnetworks that contribute significantly to the current task are achieved. This improves the accuracy and stability of pruning decisions, avoids pruning bias caused by fixed thresholds or single indicators, and thus reduces redundant expert usage while ensuring basic model performance stability.

[0076] Step S4: The edge-side control unit reconstructs the lightweight hybrid expert model based on the expert sub-networks to be retained from each layer of expert modules.

[0077] In one embodiment, physical-level model reconstruction is performed based on the expert sub-networks to be retained from each layer of expert modules. Specifically, a new model file is created, and only the weight tensors corresponding to the expert sub-networks to be retained from each layer are written into the new model file. The weight tensors of the experts to be pruned are not written into the new model file, thereby generating a smaller, lightweight model. After reconstruction, the indices of the expert sub-networks to be retained from each layer are renumbered sequentially to ensure that the subsequent routing logic of the gating network remains consistent with the simplified expert model structure, ultimately resulting in the lightweight hybrid expert model.

[0078] Step S5: During the idle window of the inference service, the edge-side control unit releases the video memory and main memory resources occupied by the full hybrid expert model, and loads the lightweight hybrid expert model to the edge-side computing acceleration unit to complete the online switching between the lightweight hybrid expert model and the full hybrid expert model.

[0079] In one embodiment, the request status of the inference service is detected in real time. When no active inference request is detected, a termination signal is sent to the original inference service process. After the original inference service process exits, the global video memory of the edge computing acceleration unit and the main memory resources of the edge control unit (CUP side) occupied by the full hybrid expert model are released.

[0080] Then, the model file loading pointer of the inference service is redirected to the lightweight hybrid expert model, a new inference service process is started, and the online model switching is completed.

[0081] In this embodiment, the model structure is reconstructed, low-frequency expert weights are removed and model files are updated in the background, and model switching is performed when preset conditions are met, realizing model updates and replacements in the running state. This achieves the actual release of redundant expert weights at the physical storage and memory levels, reduces main memory and video memory usage, reduces invalid memory accesses and cross-device data interactions, and ensures the continuity of inference services and the stability of system operation.

[0082] In one embodiment, a lightweight hybrid expert model is further employed to perform subsequent inference tasks. Since the long-tailed redundant expert subnetworks have been physically stripped, subsequent routing is only carried out within the set of retained active expert subnetworks. There is no need to allocate main memory, video memory, and bandwidth resources to redundant expert subnetworks, thereby reducing invalid memory access overhead, improving end-to-end inference throughput, and allocating the freed memory resources to key modules such as high-frequency hotspot experts or key-value caches.

[0083] In summary, the dynamic pruning method for edge-side hybrid expert model computation provided by this invention performs forward computation of the gating network during inference, bypasses the acquisition of expert indices and corresponding routing weights output by each layer of the gating network, and statistically analyzes the activation state distribution characteristics of expert modules at each layer. Based on the activation state distribution characteristics of each layer of expert modules, a comprehensive importance score vector is calculated to adaptively select the expert subnetworks to be retained for each layer of expert modules. A lightweight hybrid expert model is reconstructed based on the expert subnetworks to be retained for each layer of expert modules. The GPU and main memory resources occupied by the full hybrid expert model are released, and the online switching between the lightweight hybrid expert model and the full hybrid expert model is completed. This invention transforms the expert activation distribution characteristics generated by the hybrid expert model during the inference process of a specific task on the edge side into the basis for scheduling edge-side hardware resources, further extending the sparse computation characteristics at the model structure level into a hardware resource optimization mechanism at the edge deployment and runtime levels, providing a feasible path for the deployment of large models in edge-side hardware-constrained environments. Specifically, by physically stripping low-frequency redundant expert subnetworks from each layer of expert modules, a lightweight hybrid expert model is generated. This effectively reduces the overall storage size and runtime memory usage of the edge-side model while ensuring the core inference capabilities remain relatively stable. Simultaneously, it improves end-to-end inference efficiency and resource utilization, adapting to the limited storage capacity of edge devices and enabling efficient inference deployment in resource-constrained environments. Furthermore, after loading the lightweight model, the GPU memory usage of the edge-side computing acceleration unit and the main memory usage of the edge-side control unit are significantly reduced compared to the full model deployment mode. There is no need to reserve GPU and main memory space for long-tail ineffective experts, and the freed hardware resources can be allocated to high-frequency expert modules or key-value cache modules, further enhancing the carrying capacity of edge-side inference services.

[0084] Furthermore, it should be noted that in this method implementation, the steps are not limited to a sequential description, and the execution order is not strictly limited. Those skilled in the art can adjust the logical order of the steps according to the actual application scenario. There is no strict sequential dependency relationship, as long as it does not violate the internal logic of the technical solution.

[0085] In one embodiment, reference Figure 4 As shown, Figure 4 A schematic diagram of the overall structure of a dynamic trimming device calculated using a hybrid expert model is shown.

[0086] A dynamic cropping device 400 based on hybrid expert model computation includes:

[0087] Loading module 410 is used to load the full hybrid expert model and start the inference service, wherein the hybrid expert model includes multi-layer expert modules, and each layer of expert modules includes multiple expert sub-networks and gating networks.

[0088] The bypass acquisition module 420 is used to perform forward computation of the gating network during inference, bypass acquisition of the expert index and corresponding routing weight output by each layer of the gating network, and statistical analysis of the activation state distribution characteristics of each layer of expert modules.

[0089] The adaptive filtering module 430 is used to calculate the comprehensive importance score vector based on the activation state distribution characteristics of each layer of expert modules, so as to adaptively filter out the expert subnetworks to be retained for each layer of expert modules.

[0090] Model reconstruction module 440 is used to reconstruct a lightweight hybrid expert model based on the expert subnetworks to be retained from each layer of expert modules.

[0091] The model switching module 450 is used to release the resources occupied by the full hybrid expert model in the idle window of the inference service and load the lightweight hybrid expert model to complete the online switching between the lightweight hybrid expert model and the full hybrid expert model.

[0092] It should be noted that this device implementation can be combined with the above-described method implementation. The relevant technical details mentioned in the above method implementation remain valid in this device implementation, and will not be repeated here to avoid repetition.

[0093] In addition, embodiments of the present invention also provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the above-described dynamic pruning method for edge-side hybrid expert model computation, and achieve the same technical effect.

[0094] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described dynamic pruning method for edge-oriented hybrid expert model computation, and achieves the same technical effect.

[0095] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A dynamic pruning method for edge-end hybrid expert model computation, characterized in that, include: The full hybrid expert model is loaded into the video memory of the edge computing acceleration unit at the edge control unit and the inference service is started. The hybrid expert model includes multi-layer expert modules, and each layer of expert modules includes multiple expert sub-networks and gating networks. During inference, the edge computing acceleration unit performs forward computation of the gating network, bypasses the collection of expert indexes and corresponding routing weights output by each layer of the gating network, and statistically analyzes the activation state distribution characteristics of each layer of expert modules. Based on the activation state distribution characteristics of each layer of expert modules, a comprehensive importance score vector is calculated to adaptively select the expert subnetworks to be retained for each layer of expert modules. The edge-side control unit reconstructs a lightweight hybrid expert model based on the expert sub-networks to be retained from each layer of expert modules; During the idle window of the inference service, the edge-side control unit releases the video memory and main memory resources occupied by the full hybrid expert model, and loads the lightweight hybrid expert model to the edge-side computing acceleration unit, completing the online switching between the lightweight hybrid expert model and the full hybrid expert model.

2. The method according to claim 1, the step of bypassing the acquisition of expert indexes and corresponding routing weights output by each layer of the gating network, statistically analyzing the activation state distribution characteristics of each layer of expert modules, and storing them in the video memory of the edge computing device, comprises: A bypass activation tracing operator is deployed at the output of each layer of the gated network. The tracing operator is decoupled from the main inference path bypass and obtains the expert index and routing weight of the gated network output in read-only mode without modifying the tensor content and calculation results of the original computation graph. During the inference process, for each input token, the gating network selects Top-k experts to perform feedforward computation, and the tracking operator calls the atomic update kernel of the edge computing acceleration unit, concurrently calculates the activation frequency value and cumulative routing weight value of each expert sub-network, and stores the statistical results in the global memory of the edge computing acceleration unit. in, The activation state distribution characteristics include activation frequency value and cumulative routing weight value.

3. The method according to claim 2, characterized in that, The concurrent statistics process ends when the cumulative number of processed tokens reaches a preset trigger threshold.

4. The method according to claim 1, characterized in that, Based on the activation state distribution characteristics of each expert module, a comprehensive importance score vector is calculated, which includes: For the hybrid expert model, the first The layer expert module reads the activation frequency vector and routing weight vector of the layer, calculates the normalized distribution and distribution sharpness of the activation frequency vector, and calculates the normalized distribution and distribution sharpness of the routing weight vector; wherein, each element in the activation frequency vector and routing weight vector corresponds to an expert subnetwork, and records the cumulative activation count and routing weight value of the expert subnetwork respectively. The adaptive balance coefficient is calculated based on the distribution sharpness of the activation frequency vector and the distribution sharpness of the routing weight vector. Based on the normalized distribution of the activation frequency vector, the normalized distribution of the routing weight vector, and the adaptive balancing coefficient, the comprehensive importance score vector is calculated, expressed as: in, This is a comprehensive importance score vector, where each element corresponds to the importance score of a single expert subnetwork. For adaptive balance coefficients, Activate the frequency vector for experts; This is the routing weight vector.

5. The method according to claim 4, characterized in that, For each layer of expert modules, an expert subnetwork to be retained is adaptively selected, including: The first The elements in the comprehensive importance score vector of the layer expert module are sorted in descending order, and the current element is added to all previous elements in turn to calculate the cumulative score and the proportion of the total score. When the cumulative score and its proportion of the total score reach the preset coverage threshold, the expert subnetworks corresponding to the elements before the current element are determined as expert subnetworks to be retained, and the expert subnetworks corresponding to the remaining elements are determined as expert subnetworks to be pruned.

6. The method according to claim 1, characterized in that, Reconstruct a lightweight hybrid expert model based on the expert subnetworks to be retained from each layer of expert modules, including: Create a new model file, and only write the weight tensors corresponding to the expert subnetworks to be retained in each layer into the new model file. Do not write the weight tensors of the experts to be pruned into the new model file. The indexes of the expert subnetworks to be retained in each layer are renumbered sequentially to obtain the lightweight hybrid expert model.

7. The method according to claim 1, characterized in that, The request status of the inference service is monitored in real time. When no active inference requests are detected, a termination signal is sent to the original inference service process. After the original inference service process exits, the video memory and main memory resources occupied by the full hybrid expert model are released.

8. The method according to claim 7, characterized in that, The model file loading pointer of the inference service is redirected to the lightweight hybrid expert model, a new inference service process is started, and the online model switching is completed.

9. A dynamic cropping device based on hybrid expert model calculation, characterized in that, include: The loading module is used to load the full hybrid expert model and start the inference service. The hybrid expert model contains multiple expert modules, and each expert module contains multiple expert sub-networks and gating networks. The bypass acquisition module is used to perform forward computation of the gating network during inference, bypass acquisition of expert indexes and corresponding routing weights output by each layer of the gating network, and statistical analysis of the activation state distribution characteristics of each layer of expert modules. An adaptive filtering module is used to calculate a comprehensive importance score vector based on the activation state distribution characteristics of each layer of expert modules, so as to adaptively filter out the expert subnetworks to be retained for each layer of expert modules. The model reconstruction module is used to reconstruct a lightweight hybrid expert model based on the expert subnetworks to be retained from each layer of expert modules. The model switching module is used to release the resources occupied by the full hybrid expert model during the idle window of the inference service and load the lightweight hybrid expert model to complete the online switching between the lightweight hybrid expert model and the full hybrid expert model.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the dynamic pruning method for edge-side hybrid expert model computation as described in any one of claims 1-8.

Images