Hardware-aware joint optimization method, system, device, and medium

Abstract

The application provides a hardware-aware joint optimization method, system, device and medium. The application builds a hybrid execution mechanism between a high-level software framework and a bottom-layer hardware behavior model. The hybrid execution mechanism intercepts a gating network routing result, combines hardware behavior model simulation quantization and handling cost, jointly optimizes expert quantization bit width and affinity routing parameters, realizes collaborative minimization of precision loss and hardware overhead, generates a MoE deployment strategy adapted to hardware resource boundaries, and achieves efficiency and performance balance.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a hardware perception joint optimization method, system, device, and medium. Background Technology

[0002] Current Mixture of Experts (MoE) models generally employ a combination of routing algorithms and low-bit quantization, relying on routing constraints and weight compression to improve model performance and reduce storage overhead. Existing routing methods assign experts based on lexical features, with quantization and routing being optimized independently, and are evaluated solely using software metrics, limiting their applicability to simulation environments.

[0003] When deployed on dedicated AI chips, this solution still has many shortcomings: traditional routing does not consider hardware cache constraints, easily leading to frequent expert weight swapping, high bandwidth overhead, and high inference latency; the routing mechanism cannot perceive the hardware cache status, lacks hardware affinity scheduling, and struggles to balance model accuracy and hardware efficiency; quantization bit width and routing strategy are optimized separately, ignoring their coupling relationship and lacking a collaborative optimization mechanism under cache constraints; evaluation metrics are biased towards software dimensions and do not incorporate hardware test metrics, making chip deployment prone to cache jitter and poor adaptability. Therefore, an effective hardware-aware joint optimization method is urgently needed to solve these problems. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a hardware-aware joint optimization method, system, device and medium to overcome the above problems or at least partially solve the above problems.

[0005] To achieve the above and other related objectives, the present invention provides a hardware-aware joint optimization method, the method comprising: For the MoE layer to be optimized in the MoE model, the search agent generates a candidate joint strategy containing the quantization bit width and affinity routing parameters of each expert within a given joint search space, and then distributes the candidate joint strategy to the higher-level framework. When the high-level framework executes forward inference to the gated decision node of the MoE layer to be optimized, it calculates the original routing scores of the current lexical pair for each expert. Then, the hybrid execution engine intercepts and extracts the original routing scores at the current node, and simultaneously collects the routing decision context. The routing decision context includes the current lexical features, the original routing scores of each expert, the expert quantization bit width configuration and affinity routing parameters specified by the current candidate joint strategy, and the expert cache state maintained by the hardware behavior model. The hardware behavior model receives the routing decision context issued by the hybrid execution engine, and performs affinity routing redirection, expert transfer simulation and quantitative numerical simulation based on the virtual on-chip cache expert residency state maintained by itself, and outputs the calculation result tensor of the current node and the corresponding expert transfer cost index. After the model completes a single forward propagation, the high-level framework aggregates the computational result tensors output by each layer and accumulates various expert transfer cost indicators throughout the process. Based on this, a comprehensive evaluation result is obtained by combining the preset optimization objective function and the model accuracy cost. Then, the search agent updates the candidate joint policy parameters and iteratively optimizes them according to the comprehensive evaluation result until the preset convergence condition is met, and then outputs the target joint policy.

[0006] Optionally, before the step of generating candidate joint strategies containing the quantization bit width and affinity routing parameters of each expert within a predetermined joint search space for the MoE layer to be optimized in the MoE model, the method further includes: A pre-trained MoE model is selected and the internal MoE layers to be optimized are determined. A joint search space is constructed for each MoE layer to be optimized. The joint search space consists of a quantization bit width candidate set and an affinity routing parameter candidate set. The affinity routing parameters are used to adjust the degree of adaptation of routing decisions to the hardware cache state, including any one or more of the following: routing affinity threshold, routing reordering rules, cached expert priority coefficient, and cache hit reward coefficient. Define an optimization objective function; the optimization objective function comprehensively considers model accuracy, hardware efficiency and inference performance, including one or more of the following: model accuracy, cache miss rate, weight loading cycle, DDR read bytes, number of expert scheduling attempts, inference latency and throughput.

[0007] Optionally, for the MoE layer to be optimized in the MoE model, the search agent generates candidate joint strategies containing the quantization bit width and affinity routing parameters of each expert within a predetermined joint search space, including: Within the joint search space of each MoE layer to be optimized, the search agent samples and combines the expert quantization bit width and affinity routing parameters corresponding to each expert in the layer to generate multiple sets of heterogeneous parameter combinations. Each heterogeneous parameter combination is integrated and encapsulated to generate a one-to-one candidate joint strategy, and each candidate joint strategy is ensured to be distributed differently in the joint search space to cover different hardware adaptation and performance optimization paths.

[0008] Optionally, when the high-level framework performs forward inference to the gated decision node of the MoE layer to be optimized, it calculates the original routing scores for each expert for the current term pair; then, the hybrid execution engine intercepts and extracts the original routing scores at the current node, and synchronously collects the routing decision context, including: The high-level framework performs forward inference of the model, and when the execution flow reaches the gated decision node of the MoE layer to be optimized, it triggers the gated network to calculate the matching degree between the current word and each expert, and generates the original routing score of each expert. The hybrid execution engine intercepts the original routing score at the current node, and simultaneously collects the current lexical features, the original routing scores of each expert, the expert quantization bit width and affinity routing parameters specified by the current candidate joint strategy, and the expert cache state maintained by the hardware behavior model, and integrates them to generate a routing decision context.

[0009] Optionally, the hardware behavior model receives the routing decision context issued by the hybrid execution engine, and, based on its own maintained virtual on-chip cache expert residency state, performs affinity routing redirection, expert relocation simulation, and quantized numerical simulation, outputting the computation result tensor of the current node and the corresponding expert relocation cost index, including: The hardware behavior model receives the routing decision context issued by the hybrid execution engine, parses the original routing scores and original optimal experts corresponding to the current word, and synchronously reads the expert residency status in the current virtual on-chip cache to obtain the set of resident experts. Compare the routing scores of the original optimal expert with those of each expert in the set of resident experts. If the difference satisfies the preset affinity condition, the current word is redirected to the corresponding resident expert; otherwise, the original routing result remains unchanged. For the finally determined target expert, if it does not reside in the virtual on-chip cache, mark it as a cache miss state, estimate the weight size based on its expert quantization bit width, simulate the DMA page swapping process of loading weights from DDR to on-chip cache and swapping in and out according to the replacement rules; then perform low-bit-width numerical simulation on the target expert according to the corresponding quantization rules, and output the calculation result tensor. The simulation inference data and hardware scheduling overhead of the current MoE layer to be optimized are summarized, and the computation result tensor used for model forward propagation is output. Various expert transfer cost indicators are statistically generated, including expert loading cycle number, expert cache miss rate, DDR read bytes, expert swap-in and swap-out times, cache residency hit rate, and on-chip cache usage.

[0010] Optionally, after the model completes a single forward propagation, the higher-level framework aggregates the computational result tensors output by each layer and accumulates various expert transfer cost indicators for the entire process; based on this, a comprehensive evaluation result is obtained by combining a preset optimization objective function with the model accuracy cost solution, including: After a single forward propagation is completed in cooperation with the hardware behavior model, the high-level framework collects and summarizes the computation result tensors output by each MoE layer to be optimized, forming the output result of the single forward propagation. The expert transport cost indicators returned from each layer in the entire process of a single forward propagation are collected and accumulated synchronously to obtain the expert transport cost at the whole network level. Based on the output of a single forward propagation, the model accuracy-related indicators are calculated, and combined with the quantization error and the accuracy deviation introduced by affinity routing redirection, the overall accuracy loss cost is calculated. Substituting the overall accuracy loss cost and the expert transfer cost into the preset optimization objective function, the comprehensive evaluation result of the current candidate joint strategy is obtained.

[0011] Optionally, the step of updating the candidate joint policy parameters and iteratively optimizing them based on the comprehensive evaluation results until a preset convergence condition is met, and then outputting the target joint policy, includes: The search agent updates the quantization bit width and affinity routing parameters of each expert within it based on the comprehensive evaluation results of the current candidate joint strategies, and generates a new candidate joint strategy for the next round. A new round of hardware simulation and evaluation process is triggered based on the new candidate strategy, and the current iteration status is monitored in real time to determine whether the preset convergence conditions are met. If the preset convergence condition is not met, return to the steps of executing policy update and generation to enter the next round of iterative optimization; If the preset convergence condition has been met, the target joint strategy that adapts to the hardware resource boundary is selected from all iterations and output.

[0012] Secondly, the present invention also provides a hardware-aware joint optimization system, the system comprising: The generation module is used to generate candidate joint strategies for the MoE layer to be optimized in the MoE model. The search agent generates candidate joint strategies containing the quantization bit width and affinity routing parameters of each expert within a given joint search space, and then distributes the candidate joint strategies to the higher-level framework. The interception module is used to calculate the original routing scores of each expert for the current lexical when the high-level framework performs forward inference to the gated decision node of the MoE layer to be optimized; then the hybrid execution engine intercepts and extracts the original routing scores at the current node, and simultaneously collects the routing decision context; wherein, the routing decision context includes the current lexical features, the original routing scores of each expert, the expert quantization bit width configuration and affinity routing parameters specified by the current candidate joint strategy, and the expert cache state maintained by the hardware behavior model; The simulation module is used to receive the routing decision context issued by the hybrid execution engine from the hardware behavior model, and, based on the virtual on-chip cache expert residency state maintained by itself, to perform affinity routing redirection, expert transfer simulation and quantitative numerical simulation, and output the calculation result tensor of the current node and the corresponding expert transfer cost index. The optimization module is used to summarize the computation result tensors output by each layer after the model completes a single forward propagation, and accumulate various expert transfer cost indicators throughout the process. Based on this, a comprehensive evaluation result is obtained by combining the preset optimization objective function and the model accuracy cost. Then, the search agent updates the candidate joint policy parameters and iteratively optimizes them according to the comprehensive evaluation result until the preset convergence condition is met, and then outputs the target joint policy.

[0013] Thirdly, the present invention provides an electronic device comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program stored in the memory to cause the electronic device to perform the hardware-aware joint optimization method as described above.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the hardware-aware joint optimization method as described above.

[0015] Fifthly, the present invention provides a computer program product including computer program code, which, when run on a computer, causes the computer to implement the method described above.

[0016] The above-described one or more technical solutions provided by this invention can have the following advantages or at least achieve the following technical effects: This invention constructs a hybrid execution mechanism between a high-level software framework and a low-level hardware behavior model. The hybrid execution mechanism intercepts the gated network routing results, combines hardware behavior model simulation quantization and transportation costs, and jointly optimizes expert quantization bit width and affinity routing parameters to achieve synergistic minimization of accuracy loss and hardware overhead. This generates a MoE deployment strategy that adapts to hardware resource boundaries, achieving a balance between efficiency and performance. Attached Figure Description

[0017] Figure 1 The diagram shows a flowchart of a hardware-aware joint optimization method according to an embodiment of the present invention.

[0018] Figure 2 The diagram shows a hardware-aware MoE routing and quantization joint search architecture in one embodiment of the present invention.

[0019] Figure 3The diagram shows an affinity route redirection and expert paging mechanism in a hardware behavior model according to an embodiment of the present invention.

[0020] Figure 4 The diagram shows a data flow architecture for gated interception and bidirectional measurement collaboration in one embodiment of the present invention.

[0021] Figure 5 The diagram shows a functional module schematic of a hardware-aware joint optimization system in one embodiment of the present invention.

[0022] Figure 6 The diagram shown is a schematic representation of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0023] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0025] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0026] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0027] Unless otherwise stated, the term "multiple" means two or more.

[0028] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0029] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0030] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0031] Please see Figure 1 An embodiment of the present invention provides a hardware-aware joint optimization method, the method including the following steps S10~S40: Step S10: For the MoE layer to be optimized in the MoE model, the search agent generates a candidate joint strategy containing the quantization bit width and affinity routing parameters of each expert within a predetermined joint search space, and then sends the candidate joint strategy to the higher-level framework.

[0032] The MoE model is a neural network architecture that employs a sparse activation mechanism. It replaces traditional feedforward layers with a gated network (Router) and multiple independent expert networks. During inference, the gated network dynamically selects and activates only a few of the most relevant experts based on the input data. This "division of labor and on-demand activation" characteristic allows the model to significantly reduce the computational cost of training and inference while possessing a massive number of parameters.

[0033] The MoE layer to be optimized refers to the neural network layer in the MoE model whose operating parameters need to be determined through joint optimization; its internal expert quantization bit width and routing parameters are in an undetermined state and are used to receive candidate joint strategies for hardware simulation evaluation.

[0034] Search agents are automated control units that autonomously complete parameter combination, selection, and iterative optimization within a joint search space. They include reinforcement learning controllers, evolutionary algorithm controllers, swarm intelligence controllers, differentiable search controllers, or other automated search controllers.

[0035] The joint search space is a pre-constructed set of parameters consisting of bit-width and affinity routing parameters quantified by various experts, used to define the optimization boundary and parameter combination range for the search agent.

[0036] The expert quantization bit width is used to represent and store the numerical precision (i.e., number of bits) of the weight parameters of each expert network in the MoE layer to be optimized; it typically includes FP32 (32-bit floating point), FP16 (16-bit half-precision floating point), INT8 (8-bit integer), INT4 (4-bit integer), etc.

[0037] Affinity routing parameters refer to a set of control variables used to adjust the routing decision bias of a gated network. They include at least the routing affinity threshold, routing reordering rules, cached expert priority coefficient, and cache hit reward coefficient.

[0038] The candidate joint strategy refers to the set of parameters to be evaluated generated by the search agent in a single iteration, which includes the expert quantization bit width configuration and affinity routing parameters of each MoE layer to be optimized; it can further specify cache replacement strategy parameters, expert prefetch strategy parameters, and batch routing strategy parameters.

[0039] As an example, for the Lth MoE layer to be optimized, a candidate policy can be expressed as: E1, E2, E3, and E4 use INT4, indicating that the first four independent experts in this layer uniformly use the INT4 low-precision quantization method; E5, E6, E7, and E8 use INT8, indicating that the last four independent experts in this layer uniformly use the INT8 high-precision quantization method; affinity routing threshold. .

[0040] The high-level framework refers to the upper-level software control layer located between the model and the underlying hardware; it is responsible for coordinating model inference, policy scheduling, and data interception, specifically including controlling the model's operating logic and passing simulation parameters down to the underlying execution environment.

[0041] In practical implementation, reinforcement learning controllers, evolutionary algorithm controllers, or other automated search controllers can be selected as the optimization subject (i.e., the search agent). Then, for each MoE layer to be optimized in the MoE model, the search agent samples and combines the discrete candidate parameter sets corresponding to each expert quantization bit width and affinity routing parameter within a predefined joint search space, generating multiple sets of candidate joint strategies with differentiated distributions in batches. Each set of candidate joint strategies corresponds to different expert quantization bit width allocations and affinity routing parameter configurations to cover diverse hardware adaptation paths and performance optimization directions. Subsequently, the generated candidate joint strategies are distributed to the high-level framework so that the operator kernel and routing gating logic of the MoE layer to be optimized can be dynamically reconstructed based on the candidate joint strategies.

[0042] Step S20: When the high-level framework executes forward inference to the gated decision node of the MoE layer to be optimized, the original routing scores of each expert for the current lexical pair are calculated; then the hybrid execution engine intercepts and extracts the original routing scores at the current node, and synchronously collects the routing decision context; wherein, the routing decision context includes the current lexical features, the original routing scores of each expert, the expert quantization bit width configuration and affinity routing parameters specified by the current candidate joint strategy, and the expert cache state maintained by the hardware behavior model.

[0043] Among them, the gating decision node is the program execution location in the MoE layer to be optimized that is responsible for generating the initial routing intent and can be intercepted by the hybrid execution engine for dynamic policy optimization.

[0044] The raw routing score refers to the initial matching score vector output after the current token is input into the gated network (Router) of the MoE layer to be optimized, without any normalization or sparsification processing.

[0045] The routing decision context refers to the multi-dimensional set of states intercepted and collected by the higher-level framework at the dynamic routing optimization node of the MoE layer to be optimized. It covers the current lexical features, the original routing scores of each expert, the expert quantization bit width configuration and affinity routing parameters specified by the current candidate joint strategy, and the expert cache state maintained by the hardware behavior model.

[0046] Current lexical features refer to the high-dimensional semantic representation vector generated after the current lexical is processed by the front-end network of the MoE model (such as the attention mechanism). It is the original data basis for the MoE layer gated network to perform routing scoring and hardware affinity calculation.

[0047] Expert cache status refers to the set of metadata maintained in real time by the hardware behavior model, which reflects the dynamic attributes of the physical residence location, loading progress, and historical access popularity of each expert network in the multi-level storage medium.

[0048] Please see Figure 2 The figure illustrates a hardware-aware MoE routing and quantization joint search architecture. This architecture intercepts the original routing scores when executing the MoE gated network through a high-level framework, coordinates with the hardware behavior model to perform cache state queries, affinity redirection, and quantization simulation, and iteratively updates the deployment strategy based on the output hardware feedback metrics and accuracy costs.

[0049] In the specific implementation, when the model forward inference reaches a gated network node of a MoE layer to be optimized in the high-level framework, the original routing scores of the current lexical pair for each expert are first calculated. Then, the hybrid execution engine transparently intercepts the execution flow at the current node and simultaneously collects multi-dimensional features of the current lexical pair, the original routing scores of the current MoE layer for each expert, the expert quantization bit width and affinity routing parameters specified by the current candidate joint policy, and the expert cache state maintained in real time by the hardware behavior model. Subsequently, the above multi-dimensional information is integrated to generate a routing decision context, thereby providing a complete global state input for subsequent route redirection and dynamic scheduling.

[0050] In step S30, the hardware behavior model receives the routing decision context issued by the hybrid execution engine, and performs affinity routing redirection, expert transfer simulation and quantitative numerical simulation based on the virtual on-chip cache expert residency state maintained by itself, and outputs the calculation result tensor of the current node and the corresponding expert transfer cost index.

[0051] Virtual on-chip cache refers to the software structure within the hardware behavior model that simulates the physical on-chip cache. By tracking capacity, residency status, and replacement policies in real time, it supports affinity routing decisions and quantifies hardware overhead.

[0052] Expert residency status refers to whether the weights of each expert in MoE inference are loaded onto the computing device (including location, quantification information and timeliness) for decision-making on routing paths and optimization of resources and latency.

[0053] The computation result tensor refers to the numerical multidimensional array generated by tensor operations, which carries the intermediate or final computation results of the AI ​​model and is used for inter-layer information transmission or prediction output.

[0054] The expert swapping cost metric quantifies the hardware overhead of loading expert weights from off-chip to the computing device during MoE inference (such as the number of expert loading cycles, expert cache miss rate, DDR read bytes, number of expert swap-in and swap-out cycles, cache residency hit rate, and on-chip cache usage). It is used to optimize routing decisions and reduce resource consumption.

[0055] Expert loading cycles refer to the total number of clock cycles required to load expert weights from off-chip DDR / SSD to on-chip cache, directly reflecting the hardware latency overhead of weight transfer.

[0056] The expert cache miss rate refers to the ratio of the number of times a requested expert is not cached to the total number of requests. It is a core indicator for measuring cache efficiency.

[0057] DDR read bytes refers to the total amount of data read from main memory (DDR) for expert weights, measured in bytes (B), reflecting the bandwidth consumption of cross-level data transfer.

[0058] The number of expert swaps in and out refers to the total frequency of expert weights being swapped in and out of the cache (swap in = loaded into the cache, swap out = removed from the cache), reflecting the stability of the caching strategy.

[0059] Cache residency hit rate refers to the ratio of the number of times a requested item has resided in the cache to the total number of requests. It is the complement of the miss rate (hit rate = 1 - miss rate).

[0060] On-chip cache usage refers to the total capacity of expert weights currently occupying the on-chip cache, measured in bytes (B), reflecting the actual utilization status of hardware resources.

[0061] Please see Figure 3 This diagram illustrates the affinity-based route redirection and expert paging mechanism in the hardware behavior model. This mechanism includes two execution scenarios: Scenario A involves the original optimal expert not residing, triggering a cache miss, requiring loading from DDR and performing LRU paging; Scenario B involves the difference between the scores of the optimal and second-best experts meeting the affinity threshold, redirecting the route to the residing second-best expert for direct execution. This mechanism aims to quantify the trade-off between routing accuracy loss and hardware relocation overhead.

[0062] In the specific implementation, after the hardware behavior model receives the routing decision context, it can rely on the virtual on-chip cache expert residency status it maintains to compare the difference between the original routing scores of each expert and the scores of the experts already resided in the cache. If the difference meets the affinity threshold, the terms are dynamically redirected to experts in the cache to avoid loading overhead. Otherwise, the off-chip expert loading and replacement process is simulated and the transfer cost (such as loading cycle, DDR read / write volume, etc.) is quantified and recorded. At the same time, numerical simulation is performed on the selected expert based on the quantization bit width to generate a calculation result tensor. Finally, the expert calculation results of the current node and the corresponding expert transfer cost index are output synchronously to achieve resource optimization and hardware overhead assessment.

[0063] Step S40: After the model completes a single forward propagation, the high-level framework summarizes the computation result tensors output by each layer and accumulates various expert transfer cost indicators throughout the process; based on this, it combines the preset optimization objective function and model accuracy cost to solve the comprehensive evaluation result; then the search agent updates the candidate joint strategy parameters and iteratively optimizes them according to the comprehensive evaluation result until the preset convergence condition is met, and then outputs the target joint strategy.

[0064] The optimization objective function refers to a pre-constructed function that, based on model inference throughput, incorporates hardware constraint penalties such as bandwidth, latency, and cache misses. It quantifies the performance and resource cost of candidate strategies, guiding the search agent to iteratively optimize and balance efficiency and constraints. Its mathematical expression can be represented as:

[0065] In the formula, This represents the cost of model accuracy; This indicates the cost of expert loading; This indicates the additional precision penalty term resulting from low-bit-width quantization. This indicates the additional precision penalty term resulting from low-bit-width quantization. This represents the weighting coefficient.

[0066] The comprehensive evaluation result refers to the weighted scalar value that combines accuracy cost and hardware overhead. It is used to evaluate the quality of the strategy and guide the iterative search agent. The lower the value, the better (based on the minimization objective function design).

[0067] Preset convergence conditions refer to the pre-defined rules for determining termination during the iteration process, including reaching the maximum number of iterations, the change in the objective function being lower than a threshold, or meeting hardware constraint thresholds (accuracy / bandwidth / latency requirements). Triggering any of these conditions will terminate the iteration and output the optimal strategy that satisfies the constraints in the current search.

[0068] The joint objective strategy refers to the deployment parameter configuration that satisfies hardware constraints and has the best overall evaluation when the iteration terminates, which is the solution that balances efficiency and resources within the current search space.

[0069] Please see Figure 4 The diagram illustrates the data flow architecture of gating interception and bidirectional measurement collaboration. In a gating decision, the hybrid execution engine calls the hardware behavior model and simultaneously outputs the calculation result tensor and the expert transfer cost index. The former flows back to the high-level framework to continue forward propagation, while the latter is sent to the joint evaluation to calculate the comprehensive evaluation result, forming a search closed loop.

[0070] In the specific implementation, after the model completes the forward propagation, the high-level framework can aggregate the computation result tensors output by each MoE layer to be optimized, and accumulate the expert transfer cost indicators of the entire process (total loading cycle, DDR read volume, cache miss rate, and other network-level hardware overhead). Based on this, the accuracy cost and hardware overhead are weighted and evaluated by fusing the optimization objective function to solve for the comprehensive evaluation result. Subsequently, the search agent iteratively updates the candidate policy parameters based on the comprehensive evaluation result, optimizes the scheduling decision, until the convergence condition is met, and finally outputs the target joint policy adapted to hardware resources.

[0071] Furthermore, in one embodiment, before step S10, the method further includes steps S50-S60: Step S50: Select a pre-trained MoE model and determine the internal MoE layers to be optimized, and construct a joint search space for each MoE layer to be optimized; wherein, the joint search space consists of a quantization bit width candidate set and an affinity routing parameter candidate set; the affinity routing parameters are used to adjust the degree of adaptation of routing decisions to hardware cache state, including any one or more of the following: routing affinity threshold, routing reordering rules, cached expert priority coefficient, and cache hit reward coefficient; Step S60: Set the optimization objective function; the optimization objective function comprehensively considers model accuracy, hardware efficiency and inference performance, including one or more of the following: model accuracy, cache miss rate, weight loading cycle, DDR read bytes, expert scheduling times, inference latency and throughput.

[0072] Among them, the quantization bit width candidate set is used to independently configure differentiated quantization bit widths for different experts within the same MoE layer to be optimized, so as to achieve fine-grained heterogeneous quantization.

[0073] Affinity routing parameter candidate set is used to adjust the decision-making trade-off between the original optimal routing solution and cached resident experts in the gated network.

[0074] As an example, for each MoE layer to be optimized, the following joint search space is defined: • Expert quantization bit width candidate set: Each expert can independently choose from INT8 or INT4 quantization bit widths; • Affinity routing threshold candidate set: Let the affinity routing threshold be denoted as The candidate value range is set to [0, 0.3] or other preset intervals.

[0075] Optimize the objective function by comprehensively considering model accuracy, hardware efficiency, and inference performance, including one or more of the following: model accuracy, cache miss rate, weight loading cycle, DDR read bytes, number of expert scheduling attempts, inference latency, and throughput.

[0076] In the specific implementation, firstly, the pre-trained MoE model weights are loaded and parsed, and the model architecture is traversed to filter out the MoE layers to be optimized. Secondly, a joint search space is constructed for each MoE layer to be optimized. This joint search space includes expert-level heterogeneous quantization bit width for determining data transfer volume and affinity routing parameters for determining transfer frequency. Subsequently, an optimization objective function is established that comprehensively considers accuracy, memory transfer cost, and inference latency. The aim is to find the optimal solution in the joint search space through automated search, thereby minimizing the data transfer overhead and cache jitter of expert weights under strict constraints on model accuracy loss, and significantly improving end-to-end inference performance.

[0077] In this embodiment, a hybrid execution mechanism is constructed between the high-level software framework and the low-level hardware behavior model. The hybrid execution mechanism intercepts the gated network routing results, combines the simulation quantization and transportation costs of the hardware behavior model, and jointly optimizes the expert quantization bit width and affinity routing parameters to achieve the synergistic minimization of accuracy loss and hardware overhead. This generates a MoE deployment strategy that adapts to the hardware resource boundaries, achieving a balance between efficiency and performance.

[0078] Based on the foregoing embodiments, a second embodiment of the hardware-aware joint optimization method of the present invention is proposed. In this embodiment, step S10 may include the following sub-steps S101~S102: In sub-step S101, within the joint search space of each MoE layer to be optimized, the search agent samples and combines the expert quantization bit width and affinity routing parameters corresponding to each expert in the layer to generate multiple sets of heterogeneous parameter combinations.

[0079] Among them, heterogeneous parameter combination refers to the set of differentiated underlying configurations formed by the search agent independently matching quantization bit width and affinity routing parameters for each expert within the same MoE layer to be optimized.

[0080] In practical implementation, within the joint search space of each MoE layer to be optimized in the MoE model, the search agent can quantize the expert bit width (e.g., INT4 or INT8) and affinity routing parameters (e.g., affinity routing threshold) for each expert within the layer. Multi-dimensional random sampling or heuristic pairing is used to construct multiple sets of heterogeneous parameter combinations with differentiated characteristics. This ensures that different experts can be assigned independent quantization precision and matched with corresponding routing decision rules, providing a rich underlying parameter foundation for subsequent policy generation.

[0081] Sub-step S102 integrates and encapsulates each heterogeneous parameter combination to generate a one-to-one candidate joint strategy, and ensures that each candidate joint strategy is distributed differently in the joint search space to cover different hardware adaptation and performance optimization paths.

[0082] In practical implementation, the generated heterogeneous parameter combinations can be standardized, integrated, and encapsulated to generate one-to-one candidate joint strategies. In this process, by controlling the sampling distribution of the search agent, it is ensured that each candidate joint strategy is distributed differently in the joint search space, so as to fully cover the diverse hardware adaptation paths and performance optimization directions from high precision and low latency to low precision and high compression rate.

[0083] In this embodiment, fine-grained heterogeneous resource adaptation is achieved by flexibly sampling and combining expert quantization bit width and affinity routing parameters within the joint search space of each MoE layer to be optimized. Furthermore, discrete parameters are encapsulated into a unified candidate joint policy, establishing a direct mapping between software algorithms and hardware behaviors (such as cache hits and DDR overhead). At the same time, by leveraging the differentiated distribution of policies within the search space, it can be ensured that the search agent can broadly cover multi-dimensional performance boundaries during the initialization phase, thereby achieving a global optimization capability that balances model accuracy and hardware deployment efficiency.

[0084] Based on the foregoing embodiments, a third embodiment of the hardware-aware joint optimization method of the present invention is proposed. In this embodiment, step S20 may include the following sub-steps S201~S202: In sub-step S201, the high-level framework performs forward inference of the model, and when the execution flow reaches the gated decision node of the MoE layer to be optimized, it triggers the gated network to calculate the matching degree between the current word and each expert, and generates the original routing score of each expert.

[0085] Among them, the matching degree refers to the normalized fit probability of each expert assigned to the input lexical feature calculated by the gating network; its essence is the expert selection weight after being processed by the Softmax function, which must satisfy the probability distribution characteristics.

[0086] In the specific implementation, when the model forward inference is performed in the high-level software framework, when the execution flow reaches the gated decision node of the MoE layer to be optimized, the high-level framework can use the gated network that triggered the layer to calculate the matching degree between the current input lexical features and each expert, and generate the original routing scores of each expert, providing a basis for subsequent routing decisions.

[0087] In sub-step S202, the hybrid execution engine intercepts the original routing score at the current node, simultaneously collects the current lexical features, the original routing scores of each expert, the expert quantization bit width and affinity routing parameters specified by the current candidate joint strategy, and the expert cache state maintained by the hardware behavior model, and integrates them to generate a routing decision context.

[0088] In the specific implementation, the hybrid execution engine can intercept the execution flow at the gated decision node, and synchronously collect the current lexical features, the original routing scores of each expert, the expert quantization bit width and affinity routing parameters specified by the current candidate strategy, and the expert cache status maintained in real time by the hardware behavior model; then integrate these multi-dimensional information into a complete routing decision context to support dynamic optimization of route selection.

[0089] In this embodiment, by dynamically fusing hardware state awareness and policy constraints, the limitations of static Top-K routing in traditional MoE models are overcome. Based on the intercepted original routing scores, lexical features, expert quantization bit width, affinity parameters, and hardware cache status are integrated in real time to achieve real-time dynamic optimization of expert selection, significantly reducing communication overhead and load imbalance, and providing resource-efficient and hardware-adaptive scheduling capabilities for MoE inference.

[0090] Based on the foregoing embodiments, a fourth embodiment of the hardware-aware joint optimization method of the present invention is proposed. In this embodiment, step S30 may include the following sub-steps S301 to S304: In sub-step S301, the hardware behavior model receives the routing decision context issued by the hybrid execution engine, parses out the original routing scores and original optimal experts corresponding to the current term, and synchronously reads the expert residency status in the current virtual on-chip cache to obtain the set of resident experts.

[0091] Among them, the original optimal expert refers to the Top-1 expert output by the gating network, that is, the expert with the highest score without considering the hardware cache state.

[0092] The resident expert set refers to the subset of experts currently residing in the on-chip cache (constrained by cache capacity and replacement policy), which directly determines the cache hit / miss status.

[0093] In the specific implementation, the hardware behavior model can parse the routing decision context issued by the hybrid execution engine, extract the original routing score of the current word and the original optimal expert; and synchronously read the virtual on-chip cache state maintained by itself to obtain the set of resident experts and their quantization bit width information.

[0094] Sub-step S302: Compare the routing scores of the original optimal expert with those of each expert in the set of resident experts. If the difference satisfies the preset affinity condition, redirect the current term to the corresponding resident expert; otherwise, keep the original routing result unchanged.

[0095] Among them, the preset affinity condition refers to the pre-set threshold constraint, which is used to determine whether to redirect the lexical to the resident expert, and balance hardware efficiency and model performance by limiting the upper limit of accuracy loss.

[0096] In practical implementation, the hardware behavior model can calculate the difference in routing scores between the original optimal expert and each resident expert. If there exists a difference ≤ a preset affinity threshold, the solution is applied. If a resident expert meets the preset affinity condition, the terminology is redirected to that expert to avoid off-chip loading; otherwise, if the preset affinity condition is not met, the original routing result is maintained. This dynamically optimizes expert allocation based on the real-time cache status, ensuring that redirection only occurs when the accuracy loss is controllable (difference ≤ preset affinity threshold). This is triggered when [the time is specified], directly reducing hardware overhead.

[0097] In sub-step S303, for the finally determined target expert, if it does not reside in the virtual on-chip cache, mark it as a cache miss state, estimate the weight size according to its expert quantization bit width, simulate the DMA page swapping process of loading weights from DDR to on-chip cache and swapping in and out according to the replacement rules, and then perform low-bit-width numerical simulation on the target expert according to the corresponding quantization rules, and output the calculation result tensor.

[0098] The target expert refers to the actual execution expert determined by the routing decision, which is determined by the algorithm and the hardware state: if the original optimal expert has been cached, then the target expert is that expert; if the original optimal expert has not been cached, then the target expert is either a resident expert that meets the preset affinity conditions (redirection) or the original optimal expert that is forcibly loaded (if the conditions are not met).

[0099] The replacement rule refers to the logic that determines whether to redirect a word from the original optimal expert to a resident expert based on a preset affinity condition. It is triggered only when the original optimal expert is not resident and there is a resident expert that meets the threshold.

[0100] Quantization rules refer to the technical specifications for mapping high-precision parameters to low-precision representations in model compression. It is necessary to clarify the quantization granularity, sensitive layer processing, and error compensation strategies.

[0101] In the specific implementation, for the target expert in the final routing, the hardware behavior model can check whether it resides in the virtual on-chip cache. If it does not reside there, a cache miss is marked, and the weight is accurately estimated based on the expert's quantization bit width (e.g., INT8 / FP16). Subsequently, the DMA transfer process is simulated according to a preset cache replacement rule (e.g., LRU), including loading the weight from off-chip DDR to the on-chip cache, replacing the original resident expert, and calculating the expert loading cycle count and DDR read bytes. This achieves real-time simulation of quantized hardware overhead, outputting precise indicators (e.g., loading latency, bandwidth usage) that can guide resource scheduling through bit width-aware weight estimation and replacement strategy execution, rather than actually triggering physical transfers.

[0102] Sub-step S304: Summarize the simulation inference data and hardware scheduling overhead of the current MoE layer to be optimized, output the computation result tensor for model forward propagation, and statistically generate various expert transfer cost indicators, including expert loading cycle count, expert cache miss rate, DDR read bytes, expert swap-in and swap-out count, cache residency hit rate, and on-chip cache usage.

[0103] In the specific implementation, the dynamics of hardware resources during the forward propagation of the MoE layer to be optimized in the model can be tracked in real time. Data such as expert scheduling, cache hits and DDR access can be statistically analyzed simultaneously. The calculation result tensor and hardware overhead indicators (such as loading cycle, miss rate, read and write volume, swap-in and swap-out times, hit rate and cache usage) can be integrated into the feedback result to achieve quantitative evaluation of performance and efficiency.

[0104] In this embodiment, dynamic routing decisions and collaborative management of hardware resources are used to achieve a balance between model inference efficiency and accuracy, while providing quantitative evaluation to support performance tuning.

[0105] Based on the foregoing embodiments, a fifth embodiment of the hardware-aware joint optimization method of the present invention is proposed. In this embodiment, step S40 may include the following sub-steps S401 to S404: In sub-step S401, after the hardware behavior model completes a single forward propagation, the high-level framework collects and summarizes the computation result tensors output by each MoE layer to be optimized, forming the output result of the single forward propagation.

[0106] The output result refers to the original prediction tensor generated by the final output layer after the MoE model completes a full end-to-end inference on the input data. Its value directly reflects the model's original prediction ability for the task objective.

[0107] In a practical implementation, after the hardware behavior model simulates a single forward propagation, the higher-level framework can actively collect the output computation result tensors (such as activation values, intermediate features, etc.) from each MoE layer to be optimized. By summarizing and integrating these distributed computation results, the global output result of a single forward propagation is finally formed, providing basic data for subsequent evaluation.

[0108] Sub-step S402 involves synchronously collecting and accumulating the expert transport cost indicators returned from each layer throughout the entire forward propagation process to obtain the expert transport cost at the whole network level.

[0109] Among them, expert transfer cost refers to the quantitative indicator of hardware-level overhead caused by dynamically loading expert weights to computing devices during the MoE model inference process. Its core manifestation is the latency and bandwidth consumption caused by data transmission.

[0110] In the specific implementation, during the entire forward propagation process, each MoE layer to be optimized will return hardware behavior indicators such as expert weight loading, unloading, and cache interaction in real time (e.g., number of loading cycles, DDR read / write volume, number of cache misses, etc.). By synchronously collecting these fine-grained indicators and accumulating them layer by layer to the whole network level, the expert transport cost reflecting the hardware overhead of the entire network (e.g., total loading cycles, total DDR bytes, total miss rate) can be obtained, providing a basis for cost evaluation.

[0111] Sub-step S403 calculates the model accuracy-related indicators based on the output of a single forward propagation, and calculates the overall accuracy loss cost by combining the quantization error and the accuracy deviation introduced by affinity routing redirection.

[0112] The overall accuracy loss cost refers to the decrease in model performance relative to the ideal routing state with full accuracy caused by quantization compression and dynamic routing strategy constraints during the MoE model deployment optimization.

[0113] In the specific implementation, based on the final output of a single forward propagation, the high-level framework calculates the core indicators related to model accuracy (such as loss function value, task accuracy change, etc.); at the same time, combined with the inherent error brought by quantization and the additional accuracy deviation caused by the dynamic adjustment of affinity routing (such as prediction fluctuations introduced by route redirection), the overall accuracy loss cost of the current candidate joint strategy to model performance is comprehensively evaluated and calculated through weighted fusion or error superposition algorithms.

[0114] Sub-step S404: Substitute the overall accuracy loss cost and the expert transfer cost into the preset optimization objective function to solve for the comprehensive evaluation result of the current candidate joint strategy.

[0115] In practical implementation, the calculated overall accuracy loss cost and the network-wide expert transfer cost can be used as key inputs, substituted into a preset optimization objective function to solve, and the output scalarized comprehensive evaluation result (i.e., fitness value) is used to quantify the overall advantages and disadvantages of the accuracy and hardware overhead of the quantization strategy, providing a decision basis for the iterative optimization of the search agent.

[0116] Furthermore, in one embodiment, step S40 may further include the following sub-steps S405-S408: Sub-step S405: The search agent updates the quantization bit width and affinity routing parameters of each expert within it based on the comprehensive evaluation results of the current candidate joint strategy, and generates a new candidate joint strategy for the next round. Sub-step S406: Trigger a new round of hardware simulation and evaluation process based on the new candidate strategy, and monitor the current iteration status in real time to determine whether the preset convergence condition is met. If the preset convergence condition is not met in sub-step S407, return to the steps of executing strategy update and generation to enter the next round of iterative optimization; Sub-step S408: If the preset convergence condition has been met, select the target joint strategy that adapts to the hardware resource boundary from the previous iterations and output it.

[0117] In the specific implementation, the search agent dynamically adjusts the expert quantization bit width (e.g., high-frequency experts with low bit width improve residency, while low-frequency experts retain high-precision bit width) and affinity routing parameters (optimizing cache threshold and loading priority) based on the comprehensive evaluation results of the current strategy, generating a new candidate joint strategy that balances accuracy and hardware efficiency. Secondly, based on the new candidate joint strategy, hardware metrics such as total DDR read bytes, loading cycle, and cache misses are accumulated in real time, and a weighted fitness function (i.e., the optimization objective function) is calculated. Subsequently, the convergence status is dynamically monitored. If fitness stagnates for multiple consecutive rounds, accuracy loss exceeds limits, or hardware resources exceed limits, the iteration terminates; otherwise, optimization continues. After satisfying the preset convergence conditions, the globally optimal joint strategy is selected from historical iterations (not just the last result), and a heterogeneous quantization configuration (e.g., INT4 / FP8 mixed bit width) and hardware affinity routing parameters (e.g., cache strategy, loading queue) adapted to the hardware resource boundaries are output, generating a deployable model file to achieve Pareto optimality for accuracy and hardware efficiency.

[0118] In this embodiment, by jointly evaluating the model accuracy loss and global hardware cost, the expert quantization bit width and affinity routing parameters are dynamically optimized to achieve efficient deployment under resource constraints, balance accuracy and efficiency, ensure that quantization error and routing overhead are minimized, and achieve Pareto optimal configuration.

[0119] Based on the same inventive concept, the sixth embodiment of this invention also provides a hardware-aware joint optimization system corresponding to the hardware-aware joint optimization method of the foregoing embodiments. Since the principle of the system in the sixth embodiment of this invention for solving the problem is similar to the hardware-aware joint optimization method of the foregoing embodiments, the implementation of the system can refer to the implementation of the method; repeated details will not be elaborated further. Please refer to... Figure 5 The hardware-aware joint optimization system of the present invention may include: The generation module 10 is used to generate candidate joint strategies containing the quantization bit width and affinity routing parameters of each expert in the MoE layer to be optimized in the MoE model by the search agent in a predetermined joint search space, and to send the candidate joint strategies to the higher-level framework. The interception module 20 is used to calculate the original routing scores of the current lexical pair for each expert when the high-level framework performs forward inference to the gated decision node of the MoE layer to be optimized; then the hybrid execution engine intercepts and extracts the original routing scores at the current node, and synchronously collects the routing decision context; wherein, the routing decision context includes the current lexical features, the original routing scores of each expert, the expert quantization bit width configuration and affinity routing parameters specified by the current candidate joint strategy, and the expert cache state maintained by the hardware behavior model; The simulation module 30 is used to receive the routing decision context issued by the hybrid execution engine from the hardware behavior model, and, based on the virtual on-chip cache expert residency state maintained by itself, to perform affinity routing redirection, expert transfer simulation and quantitative numerical simulation, and output the calculation result tensor of the current node and the corresponding expert transfer cost index. The optimization module 40 is used to summarize the computation result tensors output by each layer by the high-level framework after the model completes a single forward propagation, and to accumulate various expert transfer cost indicators throughout the process; based on this, it combines the preset optimization objective function and model accuracy cost to solve the comprehensive evaluation result; then the search agent updates the candidate joint strategy parameters and iteratively optimizes them according to the comprehensive evaluation result until the preset convergence condition is met, and then outputs the target joint strategy.

[0120] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described hardware-aware joint optimization method.

[0121] Figure 6 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 6As shown, the electronic device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 6 The general will label all buses as bus systems.

[0122] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0123] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0124] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic device 400. Examples of this data include: any executable program for operation on the electronic device 400, such as the operating system 4021 and application programs 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The hardware-aware joint optimization method provided in this embodiment of the invention can be included in the application program 4022.

[0125] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the hardware-aware joint optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0126] In an exemplary embodiment, the electronic device 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0127] In summary, this invention constructs a hybrid execution mechanism between the high-level software framework and the low-level hardware behavior model. This hybrid execution mechanism intercepts gated network routing results, combines hardware behavior model simulation quantization and transportation costs, and jointly optimizes expert quantization bit width and affinity routing parameters to achieve synergistic minimization of accuracy loss and hardware overhead. This generates a MoE deployment strategy that adapts to hardware resource boundaries, achieving a balance between efficiency and performance.

[0128] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A hardware-aware joint optimization method, characterized in that, The method includes: For the MoE layer to be optimized in the MoE model, the search agent generates a candidate joint strategy containing the quantization bit width and affinity routing parameters of each expert within a given joint search space, and then distributes the candidate joint strategy to the higher-level framework. When the high-level framework executes forward inference to the gated decision node of the MoE layer to be optimized, it calculates the original routing scores of the current lexical for each expert. Then, the hybrid execution engine intercepts and extracts the original routing scores at the current node, and simultaneously collects the routing decision context. The routing decision context includes the current lexical features, the original routing scores of each expert, the expert quantization bit width configuration and affinity routing parameters specified by the current candidate joint strategy, and the expert cache state maintained by the hardware behavior model. The affinity routing parameters are used to adjust the degree of adaptation of the routing decision to the hardware cache state, including any one or more of the following: routing affinity threshold, routing reordering rules, cached expert priority coefficient, and cache hit reward coefficient. The hardware behavior model receives the routing decision context issued by the hybrid execution engine and, based on the virtual on-chip cache expert residency state maintained by itself, performs affinity routing redirection, expert transfer simulation, and quantization numerical simulation, outputting the computation result tensor of the current node and the corresponding expert transfer cost index; wherein, the expert transfer cost index refers to the hardware overhead of loading expert weights from off-chip to computing device in quantization MoE inference. After the model completes a single forward propagation, the high-level framework aggregates the computational result tensors output by each layer and accumulates various expert transfer cost indicators throughout the process. Based on this, a comprehensive evaluation result is obtained by combining the preset optimization objective function and the model accuracy cost. Then, the search agent updates the candidate joint policy parameters and iteratively optimizes them according to the comprehensive evaluation result until the preset convergence condition is met, and then outputs the target joint policy.

2. The method according to claim 1, characterized in that, Before the step of generating candidate joint strategies containing the quantization bit width and affinity routing parameters of each expert within a predetermined joint search space for the MoE layer to be optimized in the MoE model, the method further includes: A pre-trained MoE model is selected and the internal MoE layers to be optimized are determined. A joint search space is constructed for each MoE layer to be optimized. The joint search space consists of a quantization bit width candidate set and an affinity routing parameter candidate set. Define an optimization objective function; the optimization objective function comprehensively considers model accuracy, hardware efficiency and inference performance, including one or more of the following: model accuracy, cache miss rate, weight loading cycle, DDR read bytes, number of expert scheduling attempts, inference latency and throughput.

3. The method according to claim 1, characterized in that, For the MoE layer to be optimized in the MoE model, the search agent generates candidate joint strategies containing the quantization bit width and affinity routing parameters of each expert within a given joint search space, including: Within the joint search space of each MoE layer to be optimized, the search agent samples and combines the expert quantization bit width and affinity routing parameters corresponding to each expert in the layer to generate multiple sets of heterogeneous parameter combinations. Each heterogeneous parameter combination is integrated and encapsulated to generate a one-to-one candidate joint strategy, and each candidate joint strategy is ensured to be distributed differently in the joint search space to cover different hardware adaptation and performance optimization paths.

4. The method according to claim 1, characterized in that, When the high-level framework performs forward inference to the gated decision node of the MoE layer to be optimized, the original routing scores of each expert for the current word pair are calculated. The hybrid execution engine then intercepts and extracts the original routing score at the current node, and synchronously collects the routing decision context, including: The high-level framework performs forward inference of the model, and when the execution flow reaches the gated decision node of the MoE layer to be optimized, it triggers the gated network to calculate the matching degree between the current word and each expert, and generates the original routing score of each expert. The hybrid execution engine intercepts the original routing score at the current node, and simultaneously collects the current lexical features, the original routing scores of each expert, the expert quantization bit width and affinity routing parameters specified by the current candidate joint strategy, and the expert cache state maintained by the hardware behavior model, and integrates them to generate a routing decision context.

5. The method according to claim 1 or 4, characterized in that, The hardware behavior model receives the routing decision context issued by the hybrid execution engine, and, relying on its own maintained virtual on-chip cache expert residency state, performs affinity route redirection, expert relocation simulation, and quantized numerical simulation, outputting the computation result tensor of the current node and the corresponding expert relocation cost index, including: The hardware behavior model receives the routing decision context issued by the hybrid execution engine, parses the original routing scores and original optimal experts corresponding to the current word, and synchronously reads the expert residency status in the current virtual on-chip cache to obtain the set of resident experts. Compare the routing scores of the original optimal expert with those of each expert in the set of resident experts. If the difference satisfies the preset affinity condition, the current word is redirected to the corresponding resident expert; otherwise, the original routing result remains unchanged. For the finally determined target expert, if it does not reside in the virtual on-chip cache, mark it as a cache miss state, estimate the weight size based on its expert quantization bit width, simulate the DMA page swapping process of loading weights from DDR to on-chip cache and swapping in and out according to the replacement rules; then perform low-bit-width numerical simulation on the target expert according to the corresponding quantization rules, and output the calculation result tensor. The simulation inference data and hardware scheduling overhead of the current MoE layer to be optimized are summarized, and the computation result tensor used for model forward propagation is output. Various expert transfer cost indicators are statistically generated, including expert loading cycle number, expert cache miss rate, DDR read bytes, expert swap-in and swap-out times, cache residency hit rate, and on-chip cache usage.

6. The method according to claim 1, characterized in that, After the model completes a single forward propagation, the higher-level framework aggregates the computational result tensors output from each layer and accumulates various expert referral cost indicators for the entire process. Based on this, a comprehensive evaluation result is obtained by combining the preset optimization objective function with the model accuracy cost solution, including: After a single forward propagation is completed in cooperation with the hardware behavior model, the high-level framework collects and summarizes the computation result tensors output by each MoE layer to be optimized, forming the output result of the single forward propagation. The expert transport cost indicators returned from each layer in the entire process of a single forward propagation are collected and accumulated synchronously to obtain the expert transport cost at the whole network level. Based on the output of a single forward propagation, the model accuracy-related indicators are calculated, and combined with the quantization error and the accuracy deviation introduced by affinity routing redirection, the overall accuracy loss cost is calculated. Substituting the overall accuracy loss cost and the expert transfer cost into the preset optimization objective function, the comprehensive evaluation result of the current candidate joint strategy is obtained.

7. The method according to claim 1, characterized in that, The step of updating the candidate joint policy parameters and iteratively optimizing them based on the comprehensive evaluation results by the search agent until a preset convergence condition is met, and then outputting the target joint policy, includes: The search agent updates the quantization bit width and affinity routing parameters of each expert within it based on the comprehensive evaluation results of the current candidate joint strategies, and generates a new candidate joint strategy for the next round. A new round of hardware simulation and evaluation process is triggered based on the new candidate strategy, and the current iteration status is monitored in real time to determine whether the preset convergence conditions are met. If the preset convergence condition is not met, return to the steps of executing policy update and generation to enter the next round of iterative optimization; If the preset convergence condition has been met, the target joint strategy that adapts to the hardware resource boundary is selected from all iterations and output.

8. A hardware-aware joint optimization system, characterized in that, The system includes: The generation module is used to generate candidate joint strategies for the MoE layer to be optimized in the MoE model. The search agent generates candidate joint strategies containing the quantization bit width and affinity routing parameters of each expert within a given joint search space, and then distributes the candidate joint strategies to the higher-level framework. The interception module is used to calculate the original routing scores of the current lexical for each expert when the high-level framework performs forward inference to the gated decision node of the MoE layer to be optimized. Then, the hybrid execution engine intercepts and extracts the original routing scores at the current node, and simultaneously collects the routing decision context. The routing decision context includes the current lexical features, the original routing scores of each expert, the expert quantization bit width configuration and affinity routing parameters specified by the current candidate joint strategy, and the expert cache state maintained by the hardware behavior model. The affinity routing parameters are used to adjust the degree of adaptation of the routing decision to the hardware cache state, including any one or more of the following: routing affinity threshold, routing reordering rules, cached expert priority coefficient, and cache hit reward coefficient. The simulation module is used to receive the routing decision context issued by the hybrid execution engine from the hardware behavior model, and, based on the virtual on-chip cache expert residency state maintained by itself, to perform affinity routing redirection, expert transfer simulation and quantization numerical simulation, and output the calculation result tensor of the current node and the corresponding expert transfer cost index; wherein, the expert transfer cost index refers to the hardware overhead of loading expert weights from off-chip to computing device in quantization MoE inference. The optimization module is used to summarize the computation result tensors output by each layer after the model completes a single forward propagation, and accumulate various expert transfer cost indicators throughout the process. Based on this, a comprehensive evaluation result is obtained by combining the preset optimization objective function and the model accuracy cost. Then, the search agent updates the candidate joint policy parameters and iteratively optimizes them according to the comprehensive evaluation result until the preset convergence condition is met, and then outputs the target joint policy.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images