Weight processing method and device of independent graphics card architecture expert model and related equipment

Abstract

The application provides a weight processing method and device of an independent graphics card architecture expert model, electronic equipment and a readable medium. The method comprises the following steps: in an inference process of the expert model, determining at least one candidate expert and an activation probability of each candidate expert according to a routing output; updating metadata of the candidate expert according to the activation probability of the candidate expert, wherein the metadata comprises the activation probability of the candidate expert; performing a cache management operation on a storage location of a weight of the candidate expert between storage levels of VRAM, DRAM and NAND according to the metadata of the candidate expert; and performing the inference process of the expert model according to the candidate expert after the cache management operation. According to the technical solution, the candidate expert with a larger use probability can be read to a level with a faster reading speed in advance before the inference process is performed, and accurate matching with the current inference process is performed, so that inference delay and reading overhead are reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a weight processing method, apparatus, electronic device and storage medium for an expert model with a discrete graphics card architecture. Background Technology

[0002] In the field of large language model inference, the total size of all expert weights in a hybrid expert model typically far exceeds the available capacity of GPU memory and system memory on edge or local inference devices. To deploy larger-scale hybrid expert models on resource-constrained edge devices, it is usually necessary to cache the expert weights using non-volatile storage media such as high-capacity solid-state drives.

[0003] In related technologies, a common approach is to store all or most of the expert weights on a solid-state drive (SSD) and load them into video memory or RAM on demand during inference. However, this approach, which loads weights incrementally during inference, significantly increases inference latency. Furthermore, it generates numerous unnecessary random read operations from the SSD, amplifying the read amplification effect.

[0004] The aforementioned situation makes it difficult to predict and control inference latency and results in low storage bandwidth and capacity utilization efficiency when deploying large-scale hybrid expert models on edge or local devices. Therefore, how to effectively reduce random read overhead and latency during hybrid expert model inference has become an urgent technical problem to be solved. Summary of the Invention

[0005] Based on the defects and shortcomings of the existing technology, this application proposes a weight processing method, device, electronic equipment, and storage medium for an expert model of a discrete graphics card architecture.

[0006] The first aspect of this application provides a weight processing method for an expert model of a discrete graphics card architecture, comprising: during the inference process of the expert model, determining at least one candidate expert and the activation probability of each candidate expert based on the routing output; updating the metadata of the candidate expert based on the activation probability of the candidate expert, the metadata including the activation probability of the candidate expert, and the metadata also including one or more of access frequency, recent access time, and hierarchical priority; performing cache management operations on the storage locations of the candidate expert's weights among storage hierarchies composed of VRAM, DRAM, and NAND based on the candidate expert's metadata; and performing the inference process of the expert model based on the candidate expert after the cache management operation.

[0007] In one embodiment, determining at least one candidate expert and the activation probability of each candidate expert based on the routing output includes: determining experts whose activation probability evaluated by the router is greater than a preset threshold as candidate experts based on the routing output, and determining the activation probability of each candidate expert.

[0008] In one embodiment, the metadata further includes access frequency, recent access time, hierarchy priority, current inference stage, and cache capacity status; performing cache management operations on the storage locations of the candidate experts' weights among the storage tiers composed of VRAM, DRAM, and NAND based on the candidate experts' metadata includes: determining the priority score of each candidate expert based on the candidate experts' metadata; performing cache management operations on the storage locations of the candidate experts' weights among the storage tiers composed of VRAM, DRAM, and NAND based on the candidate experts' priority scores; wherein the cache management operations include one or more of prefetching, promotion, demotion, eviction, and backfilling.

[0009] In one embodiment, determining the priority score of the candidate expert based on the candidate expert's metadata includes: processing one or more of the candidate expert's activation probability, access frequency, recent access time, hierarchical priority, current inference stage, and cache capacity status using a preset scoring method to obtain the priority score of the candidate expert; wherein, the preset scoring method includes one or more of rule scoring, weighted scoring, learning predictor, least frequently used algorithm, and least recently used algorithm.

[0010] In one embodiment, the cache management operation includes prefetching and boosting. Performing cache management operations on the storage locations of the candidate experts' weights across the storage hierarchy comprised of VRAM, DRAM, and NAND based on the candidate experts' priority scores includes: if the candidate expert's priority score is greater than or equal to a first scoring threshold, then prefetching the candidate expert's weights to DRAM or VRAM; or, if the candidate expert's priority score is greater than or equal to a second scoring threshold but lower than the first scoring threshold, then boosting the candidate expert's weights from NAND to VRAM; wherein the first scoring threshold is greater than the second scoring threshold.

[0011] In one embodiment, the metadata further includes a compression state; the weight of the candidate expert from NAND to DRAM or VRAM includes: if it is determined from the compression state in the metadata that the weight of the candidate expert is stored in NAND in compressed or quantized form, then after reading the weight of the candidate expert from NAND to DRAM, decompression or dequantization is performed and the weight is then promoted to VRAM.

[0012] In one embodiment, the cache management operation includes downgrading and eviction. Performing cache management operations on the storage locations of the candidate experts' weights across storage tiers comprised of VRAM, DRAM, and NAND, based on the candidate experts' priority scores, includes: predicting the short-term reuse probability of each candidate expert based on their priority scores and historical expert access sequences; if the short-term reuse probability of a candidate expert is less than or equal to a first reuse probability threshold, and the available capacity of the storage tier containing the candidate expert's weight is lower than a capacity threshold based on the cache capacity status in the candidate expert's metadata, then downgrading the candidate expert's weight from VRAM to DRAM, or from DRAM to NAND; or, if the short-term reuse probability of a candidate expert is less than a second reuse probability threshold, and the available capacity of the storage tier containing the candidate expert's weight is lower than a capacity threshold based on the cache capacity status in the candidate expert's metadata, then eviction of the candidate expert's weight to NAND; wherein the first reuse probability threshold is greater than the second reuse probability threshold.

[0013] In one embodiment, the metadata further includes a compression state; moving the weight of the candidate expert from elimination to NAND includes: if the short-term reuse probability of the candidate expert is lower than a third reuse probability threshold, then the weight of the candidate expert is compressed and eliminated to NAND; wherein the third reuse probability threshold is less than a second reuse probability threshold.

[0014] In one embodiment, the method further includes: after performing a cache management operation, updating the metadata of each expert within the storage tier according to the cache management operation.

[0015] A second aspect of this application provides a weight processing apparatus for an expert model on a discrete graphics card architecture, comprising: a candidate expert determination unit, configured to determine at least one candidate expert and the activation probability of each candidate expert based on routing output during the inference process of the expert model; a metadata update unit, configured to update the metadata of the candidate expert based on the activation probability of the candidate expert, the metadata including the activation probability of the candidate expert, and the metadata further including one or more of access frequency, recent access time, and hierarchical priority; a cache management operation unit, configured to perform cache management operations on the storage locations of the weights of the candidate experts in a storage hierarchy consisting of VRAM, DRAM, and NAND based on the metadata of the candidate experts; and a model inference unit, configured to execute the inference process of the expert model based on the candidate experts after the cache management operation.

[0016] A third aspect of this application provides an electronic device, including a memory and a processor; the memory is connected to the processor and is used to store a program; the processor is used to implement the weight processing method of the discrete graphics card architecture expert model provided in any implementation of the first aspect by running the program in the memory.

[0017] The fourth aspect of this application provides a storage medium storing a computer program, which, when run by a processor, implements the weight processing method for the independent graphics card architecture expert model provided in any implementation of the first aspect above.

[0018] The weight processing method for expert models in discrete graphics card architecture proposed in this application determines at least one candidate expert and the activation probability of each candidate expert based on the routing output during the inference process of the expert model. This predicts the expert weights of candidate experts that may be needed in the subsequent inference process, updates the metadata of candidate experts based on their activation probabilities, and maintains metadata including at least the activation probabilities of candidate experts. The activation status of each candidate expert is updated in real time, and cache management operations are performed on the storage locations of the candidate expert weights across storage layers composed of VRAM, DRAM, and NAND using this updated metadata. This allows for the pre-reading of candidate experts with higher usage probabilities to faster access layers before the inference process, thus significantly reducing inference latency when the inference process is executed based on the cached candidate experts. Furthermore, this method retrieves expert weights across storage layers based on one or more factors, including the activation probability of the candidate weights, access frequency, recent access time, and layer priority. This fully considers the likelihood of candidate weight usage, achieving accurate matching between candidate experts and the current inference process, avoiding numerous unnecessary read operations, and reducing read overhead. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a weight processing method for an expert model of a discrete graphics card architecture, provided as an embodiment of this application.

[0021] Figure 2 This is a schematic diagram illustrating the mapping between metadata and expert status provided in one embodiment of this application.

[0022] Figure 3This is a multi-level data path diagram of the storage hierarchy provided in one embodiment of this application.

[0023] Figure 4 This is a flowchart illustrating a weight processing method for an expert model of a discrete graphics card architecture, provided as an embodiment of this application.

[0024] Figure 5 This is a flowchart illustrating a cache management operation provided in an embodiment of this application.

[0025] Figure 6 This is a schematic diagram of the architecture of an expert weighted multi-level cache management system according to an embodiment of this application.

[0026] Figure 7 This is a schematic diagram of the expert weight cache state transition according to an embodiment of this application.

[0027] Figure 8 This is a schematic diagram of an expert model weight processing device for an independent graphics card architecture according to an embodiment of this application.

[0028] Figure 9 This is a schematic diagram of a weight processing electronic device for an expert model of an independent graphics card architecture according to an embodiment of this application. Detailed Implementation

[0029] The technical solution of this application embodiment is applicable to the application scenarios of large language models, especially the application scenarios of expert models with discrete graphics card architecture, and can reduce inference latency and reading overhead.

[0030] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0031] The following is an explanation of some of the terms that may be used in this application: Mixture of Experts (MoE): A neural network architecture that uses a gating network to dynamically select a small subset of "expert" subnetworks for computation for each input.

[0032] Experts: Subnetworks or weighted reconfigurations selectively activated by routers, gating modules, or activation prediction modules in a hybrid expert model.

[0033] Token: The basic input unit for inference computation in a hybrid expert large language model. It is the smallest data granularity when text is processed by the model, typically corresponding to a word, subword, or character fragment. The hybrid expert large language model adopts a token-by-token computation process. In this process, the model does not process the entire text at once, but generates one or more subsequent tokens based on the current token, forming a streaming generation and output.

[0034] Expert weights: A set of parameters associated with a specific expert. During inference, these parameters are used to perform the actual expert calculations and generate the output.

[0035] VRAM (Video Random Access Memory) is a cache area within the GPU's video memory.

[0036] DRAM (Dynamic Random Access Memory) expert cache: Dynamic random access memory is a cache area in system memory.

[0037] NAND expert storage: Non-volatile memory, is the persistent storage area where a solid state drive (SSD) resides.

[0038] The speeds of the three media are: VRAM > DRAM > NAND.

[0039] Activation probability: represents the estimated probability that a particular expert will be selected by the router for subsequent tokens, token groups, inference layers, or request phases.

[0040] Discrete GPU architecture: This refers to the underlying logical framework of the hardware design and software collaboration within a discrete GPU chip. The "discrete GPU" in this architecture has its own Video Random Access Memory (VRAM). When using this architecture to execute hybrid expert large language model inference, the total size of all expert weights far exceeds the capacity of the GPU's VRAM and also exceeds the DRAM space reserved by the system for inference. Therefore, it is impossible to keep all expert weights permanently resident in VRAM or DRAM.

[0041] In scenarios where hybrid expert large language model inference is performed using discrete graphics card architectures, to ensure the proper functioning of models where the total number of expert weights exceeds the VRAM capacity, related technologies typically employ a combination of on-demand loading and simple caching. Specifically, the inference system temporarily stores a portion of recently used expert weights in DRAM or VRAM, while the remaining expert weights are persistently stored in the NAND flash memory of an SSD in compressed or quantized form. When the router output indicates that a particular expert is needed, the system checks whether that expert weight is already residing in VRAM or DRAM. If not, it reads the weight from the NAND flash memory into DRAM or VRAM before performing expert calculations. The main consideration for this design is to leverage the low cost and high capacity of NAND flash memory to accommodate all expert weights, while simultaneously avoiding repeated readings of the same expert weight through the caching effect of DRAM and VRAM.

[0042] However, in deployments with limited VRAM and DRAM capacity, such as edge devices, personal workstations, embedded inference devices, AI PCs, local inference devices, storage-enhanced inference servers, and system-local inference servers that combine host and SSD, the above solutions suffer from the following problems: Placing a large number of expert weights in the high-latency NAND medium to overcome capacity limitations conflicts with the requirement for low-latency acquisition of expert weights during the inference process. When the number of experts in the hybrid expert model is large and their activation is highly sparse, passively loading from NAND on request may introduce a large number of random read operations per inference step, increasing inference latency and amplifying NAND random reads. For example, during the decoding stage of a long text, the router may need to activate a different batch of experts every few steps. If the DRAM cache capacity is insufficient to accommodate these expert weights, the system needs to frequently read different expert fragments from the NAND medium, resulting in significant jitter or even persistently high inference latency. Furthermore, even if the MoE model selectively activates some experts through a routing mechanism during inference, reducing the computational load per step, the overall size of all expert weights remains large.

[0043] In addressing the issues of high inference latency and NAND random read amplification when deploying large-scale hybrid expert models with limited VRAM and DRAM capacity, the embodiments of this application determine at least one candidate expert and its activation probability based on the routing output. This allows for the prediction of the expert weights of candidate experts that may be needed in the subsequent inference process. The metadata of the candidate experts is updated based on their activation probabilities, and at least the activation probability metadata is maintained. The activation status of each candidate expert is updated in real time, and cache management operations are performed on the storage locations of the candidate expert weights across the storage hierarchy composed of VRAM, DRAM, and NAND using this updated metadata. This allows for the pre-reading of candidate experts with higher usage probabilities to faster access levels before the inference process begins, thus significantly reducing inference latency when the expert model's inference process is executed based on the cached candidate experts. Furthermore, by retrieving expert weights across storage levels based on one or more factors, including the activation probability of the candidate weights, access frequency, recent access time, and level priority, the likelihood of candidate weight usage is fully considered, achieving accurate matching between candidate experts and the current inference process, avoiding numerous unnecessary read operations, and reducing read overhead. This technology achieves the goal of efficiently executing hybrid expert large language model inference using a discrete graphics card architecture.

[0044] The expert model in this application embodiment can be, for example, a hybrid expert large language model. In this model, the router or gating mechanism selects some experts to activate during inference based on the input token, token group, request phase, or model layer, rather than performing computation on all experts. The discrete graphics card architecture in this application embodiment can be, for example, an edge device, a personal workstation, an embedded inference device, an Artificial Intelligence Personal Computer (AI PC), a local inference device, or a storage-enhanced inference server—a system architecture that cannot simultaneously maintain all expert weights of the expert model in VRAM or DRAM.

[0045] The following is combined with Figure 1 The weight processing method of the embodiments of this application will be described.

[0046] Figure 1 This is a flowchart illustrating a weight processing method for an expert model of a discrete graphics card architecture, provided as an embodiment of this application.

[0047] like Figure 1 As shown, in step S101, during the reasoning process of the expert model, at least one candidate expert and the activation probability of each candidate expert are determined based on the routing output.

[0048] Routing output refers to the expert selection information output by the router or gating module in the expert model for the current inference step. This information includes the estimated probability of each expert being activated. By obtaining the activation probability of each expert through the routing output, the system can dynamically identify candidate experts to be used during the inference process, providing a basis for decision-making for subsequent metadata updates and cache management operations.

[0049] Candidate experts refer to one or more experts identified based on routing output who may be activated in the current or subsequent inference steps. By filtering candidate experts from the routing output, the system can focus cache management operations on experts who are actually likely to participate in the computation, avoiding indiscriminate cache management for all experts and thus reducing management overhead. Activation probability represents the estimated probability that a particular expert will be selected by the router for subsequent token, token group, inference layer, or request phases. Activation probability can be derived from the router's current output, historical statistics, request similarity, hierarchical predictors, or learned prediction models. Activation probability provides the cache manager with a quantitative basis for distinguishing between hot and cold experts, enabling the system to prioritize high-probability activated experts in high-speed storage tiers and migrate low-probability activated experts to high-capacity storage tiers.

[0050] The process of determining at least one candidate expert and the activation probability of each candidate expert based on the routing output includes: determining experts whose activation probability evaluated by the router is greater than a preset threshold as candidate experts based on the routing output, and determining the activation probability of each candidate expert.

[0051] The activation probability evaluated by the router refers to the activation probability value output by the router or gating module for each expert at the current inference step. A preset threshold is used to filter the activation probabilities evaluated by the router, identifying experts with activation probabilities greater than this threshold as candidate experts. By setting a preset threshold to filter the activation probabilities evaluated by the router, the system can filter out experts with low activation probabilities, including only those with activation probabilities exceeding the preset threshold in the candidate range. This concentrates cache management operations on experts who are truly likely to participate in the computation, reducing unnecessary data migration and management overhead. The introduction of the preset threshold allows the system to flexibly adjust the strictness of candidate expert selection according to deployment requirements—a higher threshold results in fewer candidate experts, making cache management operations more focused, suitable for scenarios with extremely limited storage resources; a lower threshold provides a wider range of candidate experts, helping to improve cache hit rate, suitable for scenarios with relatively abundant storage resources.

[0052] Step S102: Update the metadata of the candidate experts according to their activation probability. The metadata includes the activation probability of the candidate experts, as well as one or more of the following: access frequency, most recent access time, and hierarchical priority.

[0053] Metadata is cache management information maintained for each expert or expert shard. By maintaining multi-dimensional expert-level metadata, the system can comprehensively consider factors such as expert activation probability, historical access frequency, and hierarchical priority when making caching decisions. Compared to strategies that rely on only a single dimension (such as relying solely on activation probability), this approach can more accurately reflect the actual value of expert weights, thereby improving cache hit rate. In one embodiment, the metadata fields are shown in Table 1.

[0054] Table 1. Examples of metadata fields Access frequency refers to the number of times an expert has been accessed recently, or a weighted count. This identifies frequently used experts and prevents them from being prematurely demoted to lower-speed storage tiers, thus reducing redundant loading overhead. Recent access time refers to the time when an expert was last predicted, loaded, hit, or executed. This allows the system to identify long-unaccessed "cold" experts and prioritize migrating them to lower-speed storage tiers when storage capacity is limited, thereby freeing up high-speed storage space. Tier priority represents the preferred cache tier for an expert. Tier priority can be determined based on activation probability, access history, latency sensitivity, model layer location, and cache pressure. Tier priority protects the weights of highly probable experts from being downgraded or eliminated under capacity pressure, ensuring they remain in high-speed storage tiers. The mapping between metadata and expert status is detailed below. Figure 2 .like Figure 2 As shown, metadata can be maintained based on access statistics / hierarchical priority, NAND physical location, and activation profile. Hierarchical mapping in the metadata can be maintained based on hierarchical priority, weight format status, and migration and operation trajectory, and used as input for policy decision.

[0055] Step S103: Perform cache management operations on the storage locations of the candidate experts' weights in the storage hierarchy consisting of VRAM, DRAM, and NAND, based on the candidate experts' metadata.

[0056] The storage hierarchy consisting of VRAM, DRAM, and NAND is a multi-level cache system: the VRAM layer is fast but has a small capacity, used to store expert weights that are available immediately; the DRAM layer has a moderate speed and capacity, used to temporarily store expert weights for promotion to VRAM or to retain expert weights that have been downgraded from VRAM; the NAND layer is persistent, high-capacity storage used to store the weights of low-frequency experts. This three-level storage hierarchy allows the system to distribute expert weights across storage media of different speeds and capacities based on their access frequency, accommodating larger-scale expert models within the limited capacity of VRAM and DRAM, while reducing inference latency by keeping hot expert weights in high-speed layers.

[0057] It's important to note that expert weights are a set of parameters corresponding to a specific expert. Expert weights can be further divided at the granularity of model layers, expert fragments, tensor blocks, precision formats, or storage objects. This multi-granularity division of expert weights allows the system to perform cache management operations at a finer granularity, such as prefetching only a specific layer fragment of a particular expert instead of the entire expert weight, thereby improving storage space utilization and transmission efficiency.

[0058] By coordinating expert-level metadata with multi-level storage tiers, the system can prioritize keeping high-value expert weights in high-speed storage tiers and store low-frequency expert weights in high-capacity storage tiers, thereby improving cache hit rate, reducing redundant loading and random NAND read overhead, and reducing inference latency.

[0059] Step S104: Perform the reasoning process of the expert model based on the candidate experts after the cache management operation.

[0060] Figure 3 This is a multi-level data path diagram of the storage hierarchy provided in one embodiment of this application. For example... Figure 3 As shown, during expert model inference, there are multiple data paths across the three layers of VRAM, DRAM, and NAND. It can be determined whether each candidate expert resides in VRAM, is temporarily stored in DRAM, or is only stored in NAND. For a VRAM hit, the token is distributed to the corresponding expert for computation. For a DRAM hit, the expert weight is promoted from DRAM to VRAM, provided the activation probability and capacity strategy allow. For experts residing only in NAND, the expert weight is read from the NAND medium, decompressed or dequantized if necessary, and the weight is temporarily stored or promoted.

[0061] In one embodiment, after performing cache management operations, the metadata of each expert within the storage tier can also be updated according to the cache management operations.

[0062] The weight processing method for expert models in discrete graphics card architecture proposed in this application determines at least one candidate expert and the activation probability of each candidate expert based on the routing output during the inference process of the expert model. This predicts the expert weights of candidate experts that may be needed in the subsequent inference process, updates the metadata of the candidate experts based on their activation probabilities, and maintains metadata including at least the activation probabilities of the candidate experts. Using this updated metadata, cache management operations are performed on the storage locations of the candidate expert weights across storage layers composed of VRAM, DRAM, and NAND. This allows for the pre-reading of candidate experts with higher usage probabilities to faster access layers before the inference process, thus significantly reducing inference latency when the expert model's inference process is executed based on the cached candidate experts. Furthermore, this method retrieves expert weights across storage layers based on activation probability, access frequency, recent access time, and layer priority, fully considering the likelihood of candidate weight usage and achieving accurate matching between candidate experts and the current inference process, avoiding numerous unnecessary read operations and reducing read overhead.

[0063] Figure 4 This is a flowchart illustrating a weight processing method for an expert model of a discrete graphics card architecture, provided as an embodiment of this application.

[0064] like Figure 4 As shown, in one embodiment, the metadata also includes access frequency, recent access time, hierarchy priority, current inference stage, and cache capacity status.

[0065] The current inference stage refers to the current execution stage in the expert model's inference process, such as the pre-filling stage or the decoding stage. The access patterns for expert weights may differ across inference stages. For example, the pre-filling stage may require a large number of different experts to participate in the computation, while in the decoding stage, consecutive tokens may repeatedly activate the same expert. The system can adjust its prefetching and retention strategies accordingly. Cache capacity status refers to the current available capacity or occupancy information for each storage tier, including VRAM, DRAM, and NAND. Incorporating cache capacity status into metadata enables the system to promptly trigger degradation or eviction operations when storage tier capacity is strained, and appropriately relax retention strategies when capacity is ample, thereby achieving dynamic balancing of storage resources.

[0066] Step S101: During the inference process of the expert model, determine at least one candidate expert and the activation probability of each candidate expert based on the routing output.

[0067] Step S102: Update the metadata of the candidate experts according to their activation probability. The metadata includes the activation probability of the candidate experts, as well as one or more of the following: access frequency, most recent access time, and hierarchical priority.

[0068] Step S1031: Determine the priority score of each candidate expert based on the candidate expert's metadata.

[0069] Priority scoring is a quantified score calculated based on multiple dimensions of parameters in the metadata of candidate experts. It is used to quantify the caching value of candidate experts. Priority scoring integrates multi-dimensional metadata information into a single comparable value, enabling the cache manager to quickly determine the relative priority of each candidate expert when prefetching, promotion, demotion, or eviction operations are required, avoiding decision conflicts between multi-dimensional information. Based on the priority scores of candidate experts, the system performs cache management operations on the storage locations of candidate experts across the storage hierarchy composed of VRAM, DRAM, and NAND, including one or more of prefetching, promotion, demotion, eviction, and backfilling. Figure 5 This is a flowchart illustrating a cache management operation provided in one embodiment of this application. Figure 5 As shown, the process revolves around candidate expert identification, metadata query, cache decision-making, expert execution, and subsequent metadata maintenance in the current inference step.

[0070] During the inference process of the expert model, the first stage is the Token Batch stage. This stage corresponds to one or more tokens pending processing in the current request or decoding step. The system triggers routing calculations based on the current token batch to determine the experts that may be activated during this inference process.

[0071] Subsequently, the system enters the routing output phase. Based on the input characteristics of the current token batch, the router outputs one or more candidate experts and their corresponding activation tendency information. Based on this routing output, the system can determine the candidate experts that may participate in the calculation during this inference process, thus providing a basis for subsequent cache scheduling.

[0072] After identifying candidate experts, the system enters the metadata query phase. In this phase, the cache manager queries the metadata corresponding to each candidate expert, checking the current storage tier of the candidate expert and its status information related to caching decisions. This metadata may include, for example, activation probability, access frequency, last access time, compression status, and tier priority. By querying the metadata, the system can determine whether the weight of a candidate expert currently resides in VRAM, DRAM, or is only stored in NAND, and further evaluate its caching value.

[0073] Based on the metadata query results, the system enters the cache decision-making stage. In this stage, the system combines the cache hit status, the activation probability of candidate experts, and the current capacity status to make decisions such as hit, prefetch, or load on the weights of candidate experts. Figure 5 Several typical decision branches are shown in the figure.

[0074] Firstly, if a candidate expert is a high-activation-probability expert, the system can prefetch the expert's weight into VRAM or DRAM before the corresponding expert layer arrives. By performing prefetching on high-activation-probability experts, the waiting time during inference execution can be reduced, allowing the weights of high-value experts to reside in a faster storage layer as early as possible.

[0075] Secondly, if a candidate expert fails to reach a VRAM target but succeeds in a DRAM target, it means that the expert's weight is not currently residing in VRAM but is temporarily stored in DRAM. In this case, the system can directly promote the expert's weight from DRAM to VRAM and simultaneously update the corresponding residency status metadata. Compared to rereading the weight from NAND, this method can complete the expert weight readiness with lower latency.

[0076] Third, if a candidate expert fails to be found in either VRAM or DRAM, it means that the expert weight is not currently residing in any high-speed or medium-speed cache level, but is only stored in NAND. For such candidate experts, the system can read the corresponding compressed or quantized expert weights from NAND, and perform decompression or dequantization processing when needed, then load them into DRAM or further up to VRAM for use in subsequent expert calculations.

[0077] Fourth, if a candidate expert is a low-activation-probability expert, the system can choose to keep the expert's weight resident in the NAND and delay loading. By not immediately loading or promoting low-activation-probability experts, the limited VRAM and DRAM resources can be avoided from being occupied by low-value experts for a long time, thus reserving space for higher-priority experts.

[0078] After completing the caching decision and preparing the corresponding expert weights, the system enters the expert execution phase. In this phase, candidate experts with available weights are used to perform the corresponding expert calculations. The MoE execution engine reads the corresponding expert weights from VRAM, performs inference on the token assigned to that expert, and thus completes the expert calculation process.

[0079] During or after expert execution, the system also performs post-processing related to cache maintenance. The first step is the metadata update process. For experts that have been accessed, loaded, promoted, or executed, the system updates their corresponding metadata, including access frequency, last access time, activation probability, compression status, and priority level. By updating metadata in a timely manner, it ensures that subsequent caching decisions are based on the latest state.

[0080] After the metadata update, the system can further perform a capacity pressure check. This process assesses the current capacity levels of VRAM and DRAM to determine if there is a cache shortage. When insufficient available VRAM or DRAM capacity is detected, the system triggers subsequent migration and release mechanisms.

[0081] When capacity pressure is high, the system enters a degradation and retirement phase. During this phase, the system migrates low-priority experts from higher-speed storage tiers to lower-speed storage tiers based on expert ratings, access history, or short-term reuse probability. For example, experts residing in VRAM can be migrated to DRAM, or further to NAND, to free up limited cache space.

[0082] After degradation or eviction is completed, the system can also perform a backfill operation. The backfill operation is used to persistently save the updated NAND metadata or the recompressed expert weights to ensure that the weight copies and metadata in the NAND remain consistent with the current cache state. Through the backfill mechanism, the system can support subsequent reloading, recovery, and consistency maintenance.

[0083] pass Figure 5 As shown in the process, the system can prioritize ensuring that the weights of experts with high activation probability are ready in a timely manner before or during inference, and perform downgrading, elimination and backfilling of low-value experts when capacity is limited, thereby reducing inference latency and reducing NAND random read overhead.

[0084] In an exemplary embodiment, a preset scoring method can be used to process one or more of the following factors of a candidate expert: activation probability, access frequency, recent access time, hierarchical priority, current inference stage, and cache capacity status, to obtain a priority score for the candidate expert. The preset scoring methods include one or more of the following: rule-based scoring, weighted scoring, learned predictors, Least Frequently Used (LFU) algorithm, and Least Recently Used (LRU) algorithm. By providing a variety of scoring methods, the system can flexibly select the most suitable scoring strategy based on different deployment scenarios and computing resource constraints.

[0085] Rule-based scoring refers to determining the priority score of candidate experts by judging metadata fields based on predefined rules or conditions. Rule-based scoring is simple to implement, has low computational overhead, and is suitable for edge devices with limited computing resources, enabling caching decisions with relatively low additional overhead. Weighted scoring assigns weight coefficients to multiple fields in metadata and calculates the priority score of candidate experts through weighted summation or weighted combination. Weighted scoring allows for flexible adjustment of the contribution of each metadata dimension to the final score, enabling the system to be customized according to the focus of different application scenarios (such as prioritizing activation probability or access frequency). Learning predictors use machine learning models to predict the priority score of candidate experts based on metadata and historical access patterns. Learning predictors can automatically learn complex patterns of expert activation from historical data, capturing non-linear relationships that are difficult to express with rule-based and weighted scoring, thus providing more accurate priority scores in scenarios with complex expert activation patterns. The LFU algorithm is an algorithm that makes cache replacement decisions based on access frequency, prioritizing the elimination of experts with the lowest access frequency. It can protect the weight of frequently accessed experts from being eliminated and is suitable for scenarios where expert activation patterns are relatively stable and some experts are continuously used at high frequency. The LRU algorithm is an algorithm that makes cache replacement decisions based on the most recently accessed time. It prioritizes the removal of experts who have been accessed the least recently and can quickly remove cold experts who have not been accessed for a long time. It is suitable for scenarios where the expert activation mode changes dynamically with the inference process.

[0086] The aforementioned preset scoring methods can be one or a combination of multiple methods. For example, LRU can be combined with activation probability prediction, or LFU can be combined with weighted scoring. Combining multiple scoring methods allows the system to simultaneously consider cache value judgments from different dimensions, compensating for the limitations of a single strategy, and thus achieving better cache management performance in complex inference scenarios.

[0087] Step S1032: Perform cache management operations on the storage locations of the candidate experts in the storage hierarchy consisting of VRAM, DRAM and NAND according to the priority scores of the candidate experts; wherein, the cache management operations include one or more of prefetching, promotion, demotion, eviction and backfilling.

[0088] The specific execution methods of prefetching, promotion, elimination, downgrading, and backfilling mentioned in this embodiment are shown in Table 2.

[0089] Table 2 Cache Management Operations The following describes the various cache management operations. Prefetching refers to the operation of loading expert weights from a low-speed storage layer to a high-speed storage layer before they are actually needed. For example, before the corresponding expert layer is executed, expert weights are read from NAND to DRAM or promoted from DRAM to VRAM. Prefetching ensures that weights with a high probability of activating an expert are ready in the high-speed storage layer before computation begins, avoiding delays caused by waiting for weight loading during inference. Promotion refers to the operation of copying or decoding expert weights from a lower-speed storage layer to a higher-speed storage layer. For example, copying expert weights from DRAM to VRAM. Promotion allows warm expert weights already temporarily stored in DRAM to quickly enter VRAM when needed, significantly shortening weight readiness time compared to reloading from NAND.

[0090] Degradation refers to the operation of moving expert weights from a higher-speed storage tier to a lower-speed storage tier. For example, moving expert weights from VRAM to DRAM. Degradation frees up high-speed storage space when VRAM capacity is limited, while retaining expert weights in DRAM for near-term reuse, avoiding the high latency overhead of directly evicting them to NAND and then reloading. Elimination refers to releasing cached copies of expert weights in VRAM or DRAM, retaining only valid copies in NAND. Elimination frees up high-speed storage space when storage capacity is severely insufficient, ensuring that higher-value expert weights reside in high-speed tiers. Backfilling refers to writing updated weight copies or metadata back to NAND and logging the process after expert weights have been downgraded, eliminated, recompressed, requantized, or rearranged. Backfilling ensures that the expert weight copies and metadata in NAND remain consistent with the current system state, enabling the correct weight data and format information to be obtained when reloading from NAND subsequently.

[0091] In one exemplary embodiment, the cache management operation may include prefetching and boosting. Step S1032 may further include: if the priority score of a candidate expert is greater than or equal to a first scoring threshold, then prefetching the weight of the candidate expert to DRAM or VRAM. Alternatively, if the priority score of a candidate expert is greater than or equal to a second scoring threshold but lower than the first scoring threshold, then boosting the weight of the candidate expert from NAND to VRAM; wherein the first scoring threshold is greater than the second scoring threshold.

[0092] Candidate experts whose priority scores reach the first scoring threshold are considered to have a high probability of activation. Their weights are prefetched into DRAM or VRAM so that the expert weights are ready in the high-speed storage level before the calculation begins, avoiding waiting delays during the inference process.

[0093] Candidate experts whose priority scores fall between the second and first score thresholds have a moderate activation probability. Their weight is increased from NAND to VRAM, allowing them to directly participate in computation when actually activated, while avoiding the storage pressure of performing prefetch operations on all candidate experts. Furthermore, setting the first score threshold higher than the second score threshold ensures that prefetch operations are performed only on the highest-value candidate experts, while boosting operations cover the next highest-value candidate experts, forming a hierarchical cache management strategy to prevent high-speed storage tiers from being filled by too many candidate experts.

[0094] The prefetching operation mentioned above can be triggered by routing output, activation predictors, request-level expert activation trajectories, layer execution plans, or timed windows. These multiple triggering methods allow the prefetching operation to adapt to different inference scenarios. For example, for an upcoming model layer, the system can prioritize prefetching high-probability activation experts for that layer; for consecutive tokens in the decoding phase, the system can pre-reserve experts for reuse based on recent activation sequences.

[0095] In one exemplary embodiment, the metadata also includes a compression state. This compression state is the current storage format information of the expert weights recorded in the metadata, and may represent, for example, the original state, compressed state, quantized state, decompressed state, or dequantized state. The compression state enables the cache manager to automatically determine whether decompression or dequantization needs to be performed during the boosting operation, avoiding the direct use of unrestored weights for calculation, which could lead to incorrect inference results. In this embodiment, boosting the weights of candidate experts from NAND to DRAM or VRAM includes: If the weights of candidate experts are determined to be stored in NAND in compressed or quantized form based on the compression status in the metadata, then the weights of candidate experts are read from NAND to DRAM, decompressed or dequantized, and then promoted to VRAM.

[0096] In this process, DRAM serves as an intermediate buffer layer, carrying compressed or quantized data read from NAND. After format restoration in DRAM, weights meeting execution accuracy requirements are then promoted to VRAM. Using DRAM as an intermediate buffer layer for decompression or dequantization avoids the memory bandwidth and computational resources consumed by format restoration operations in VRAM, while ensuring that weights promoted to VRAM are in a format directly usable for computation. Compression methods can include general compression, sparse coding, or other weight compression formats. Storing low-frequency expert weights in compressed form significantly reduces storage space usage in NAND, allowing limited NAND capacity to accommodate more expert weights. Quantization methods can include weighted quantization or grouped quantization. Storing expert weights in quantized form reduces storage space usage while allowing dequantization to restore the data to a format meeting inference accuracy requirements, thus improving storage efficiency and ensuring inference accuracy.

[0097] In an exemplary embodiment, the cache management operation includes degradation and eviction. Step S1032 may further include: predicting the short-term reuse probability of each candidate expert based on the priority score of the candidate experts and the historical expert access sequence. The short-term reuse probability represents the probability that each candidate expert will be reactivated in a short period of time. By combining the priority score and the historical expert access sequence for prediction, the short-term reuse probability can more accurately reflect the possibility of candidate experts being reused in recent inference steps. Compared with the judgment method that relies solely on static metadata, it can capture the dynamic temporal pattern of expert activation. The historical expert access sequence refers to the historical sequence information of expert activation recorded during the inference process, which provides temporal context information for the prediction of the short-term reuse probability, enabling the system to identify the sequence pattern of expert activation (e.g., some experts tend to be continuously activated after a certain expert), thereby improving the accuracy of the reuse probability prediction.

[0098] If the short-term reuse probability of a candidate expert is less than or equal to a first reuse probability threshold, and the available capacity of the storage tier containing the candidate expert's weight is lower than a capacity threshold based on the cache capacity status in the candidate expert's metadata, then the candidate expert's weight will be downgraded from VRAM to DRAM, or from DRAM to NAND. Alternatively, if the short-term reuse probability of a candidate expert is less than a second reuse probability threshold, and the available capacity of the storage tier containing the candidate expert's weight is lower than a capacity threshold based on the cache capacity status in the candidate expert's metadata, then the candidate expert's weight will be removed from the storage tier and moved to NAND. The first reuse probability threshold is greater than the second reuse probability threshold.

[0099] The downgrade operation is triggered under the dual conditions of low short-term reuse probability of candidate experts and tight storage tier capacity. It migrates the expert weight to the adjacent low-speed tier instead of directly eliminating it to NAND, so that the expert weight can be quickly recovered from the higher-speed tier when it is recently reactivated, reducing the reload latency.

[0100] The elimination operation is triggered when the probability of short-term reuse of candidate experts is extremely low and the storage level capacity is tight. It directly releases the cached copies in VRAM or DRAM, and retains only the valid copies in NAND, thereby maximizing the release of high-speed storage space for higher-value expert weights.

[0101] The first reuse probability threshold is greater than the second reuse probability threshold. This approach forms a hierarchical migration strategy: candidate experts with short-term reuse probabilities between the two thresholds are only downgraded to the adjacent low-speed level, while candidate experts with short-term reuse probabilities below the second reuse probability threshold are directly eliminated to NAND. This hierarchical mechanism avoids an indiscriminate elimination strategy for all low reuse probability experts, achieving a balance between freeing up storage space and preserving reuse potential.

[0102] The capacity threshold mentioned above is used to determine whether the available capacity of the storage tier containing the candidate expert weights is strained. The system only triggers degradation or eviction operations when the available capacity falls below the capacity threshold. The introduction of the capacity threshold ensures that degradation and eviction operations are only executed when the storage tier is indeed facing capacity pressure, avoiding unnecessary data migration when storage resources are plentiful, and reducing transmission overhead and inference latency jitter.

[0103] The multi-level cache structure of this embodiment may include a faster but smaller VRAM layer, a DRAM layer with moderate speed and capacity, and a persistent and larger NAND layer. Table 3 lists exemplary cache levels and storage objects.

[0104] Table 3 Cache Levels and Storage Objects In one exemplary embodiment, the metadata also includes a compression status. Moving the weights of candidate experts from NAND flash memory includes: if the short-term reuse probability of a candidate expert is lower than a third reuse probability threshold, then the weights of the candidate expert are compressed and moved to NAND flash memory; wherein the third reuse probability threshold is lower than a second reuse probability threshold. The setting that the third reuse probability threshold is lower than the second reuse probability threshold ensures that the compression and removal operation is performed only when the short-term reuse probability of a candidate expert is extremely low. This setting avoids unnecessary compression of expert weights with moderate reuse probabilities—because compressed weights require additional decompression overhead when subsequently reactivated. Performing compression only on experts with extremely low reuse probabilities achieves a reasonable balance between storage space savings and subsequent decompression overhead.

[0105] Compressing and discarding candidate expert weights before writing them to NAND refers to compressing the weight data before writing it to NAND during the process of discarding candidate expert weights from VRAM or DRAM. Compressed discarding allows cold expert weights to be stored in a more compact form in NAND, reducing NAND storage space usage and the amount of data written during subsequent backfilling, while also reducing NAND write wear. Compression can be, for example, general compression, sparse coding, or other weight compression methods. In some embodiments, quantization can be used instead of compression, discarding weights to NAND in quantized form.

[0106] Step S104: Perform the reasoning process of the expert model based on the candidate experts after the cache management operation.

[0107] In one exemplary embodiment, after performing cache management operations, the metadata of each expert within the storage tier can also be updated according to the cache management operations. The update method ensures that the metadata always reflects the latest storage location, format status, and access information of each expert's weight. Timely updates to metadata enable subsequent cache management decisions to be based on accurate status information, avoiding erroneous prefetching, duplicate loading, or invalid eviction due to outdated metadata. This update includes, but is not limited to, updating metadata fields such as the expert's residency status, tier priority, last access time, compression status, and NAND physical location. Synchronous updates to multiple metadata fields ensure that the cache manager can obtain complete and consistent status information for each expert's weight in the next decision, avoiding decision bias caused by outdated fields.

[0108] This update can be implemented through an operation log and a backfill queue. The operation log records prefetch, promotion, demotion, eviction, backfill, and failure recovery events; the backfill queue is used to asynchronously persist metadata or recompress weights to NAND media. Implementing metadata updates through the operation log and backfill queue allows the system to asynchronously persist metadata without blocking the main inference flow, reducing the impact of metadata updates on inference latency.

[0109] Through metadata-driven cache management, the system can dynamically adjust the storage location of expert weights during inference, migrating high-value expert weights to high-speed levels and low-value expert weights to high-capacity levels, thus achieving efficient expert weight management under conditions of limited storage resources. After the cache management operation is completed, the system executes the inference process of the expert model based on the candidate experts after the cache management operation, ensuring that the expert weights involved in the calculation are ready in the high-speed storage level.

[0110] Figure 6 This is a schematic diagram of the architecture of an expert weighted multi-level cache management system according to an embodiment of this application.

[0111] The system includes an inference request input module, a routing and activation prediction module, an expert cache manager, a MoE execution engine, and a multi-level cache storage system. The inference request input module receives the input tokens and session state to be processed and passes the request to the routing and activation prediction module. The routing and activation prediction module generates candidate expert identifiers and corresponding activation probabilities based on the model's internal routing mechanism and sends this information to the expert cache manager.

[0112] The multi-level cache storage system consists of three levels: VRAM expert cache, DRAM expert cache, and NAND expert storage. The VRAM expert cache resides in the graphics processor's video memory and stores uncompressed expert weights that can be directly used by the MoE execution engine. The DRAM expert cache resides in system memory and temporarily stores warm expert weights or recently downgraded weights from VRAM, preparing them for promotion to VRAM. The NAND expert storage resides in the flash memory of the solid-state drive and preferably stores low-frequency expert weights in compressed or quantized form, providing large-capacity persistent storage.

[0113] The expert cache manager is connected to the expert metadata table. The expert metadata table records cache metadata for each expert or expert shard, including access frequency, last access time, activation probability, hierarchical priority, compression status, and NAND physical location. The policy scheduler retrieves information from the expert metadata table and provides decision recommendations to the expert cache manager based on cache hit targets, latency budgets, and space constraints.

[0114] The transmission and decoding engine connects between the DRAM expert cache and the VRAM expert cache, and also connects the NAND expert storage and the DRAM expert cache. It is responsible for performing direct memory access transmission, decompression, dequantization, and consistency checks between different storage levels. The operation log and backfill queue connect to the DRAM expert cache and the host or SSD-side control module to record the trajectory of migration, eviction, and backfill operations, and drive asynchronous persistent updates. The physical layout index connects to the NAND expert storage and the host or SSD-side control module to map expert fragments to block, page, and version information in the NAND flash memory to support fast retrieval. The host or SSD-side control module coordinates the execution location of the caching strategy; depending on deployment needs, some decision-making logic can be configured to be executed on the host side, the SSD side, or both collaboratively.

[0115] In this architecture, the MoE execution engine is directly connected to the VRAM expert cache in the multi-level cache storage system. Once the weights of the candidate experts are ready, the MoE execution engine retrieves the weight data from the VRAM expert cache and completes the calculation of the hybrid expert model.

[0116] Figure 7 This is a schematic diagram of the expert weight cache state transition according to an embodiment of this application. The following is in conjunction with… Figure 7 The process of expert weights transitioning between storage layers consisting of VRAM, DRAM, and NAND is explained in detail.

[0117] like Figure 7 As shown, the lifecycle of expert weights can be divided into several states, and the cache manager drives the transition between states based on metadata.

[0118] Expert weights are initially stored in NAND flash memory in compressed or quantized form, in a NAND-only state. After the system determines candidate experts based on routing output, the expert cache manager queries the expert metadata table and calculates the priority score for each candidate expert based on metadata such as activation probability, access frequency, recent access time, and hierarchical priority. For high-value candidate experts whose priority scores reach the first threshold, the system triggers a prefetch operation, reading the expert's weight from NAND to DRAM. At this point, the expert weight enters a DRAM temporary storage state. The DRAM temporary storage state is an intermediate state for expert weights migrating from low-speed persistent storage to high-speed video memory. In this state, the weight can wait for decompression, dequantization, or be used directly by subsequent operations.

[0119] If expert weights are prefetched from NAND to DRAM in compressed or quantized form, the system performs decompression or dequantization operations within DRAM based on the compression status recorded in the metadata, restoring them to a format that meets the accuracy requirements of inference execution. After format restoration, or for expert weights temporarily stored in DRAM in their original format, when the inference execution unit needs to schedule the expert, the system performs a promotion operation on the expert weight, copying or migrating it from DRAM to VRAM. At this point, the expert weight enters a VRAM-resident state. Expert weights in the VRAM-resident state can be directly accessed by the MoE execution engine without any additional format conversion or data migration.

[0120] When the computation of a corresponding expert is triggered, the MoE execution engine distributes the relevant token or computation task to that expert, and the expert weights enter an active computation state. In the active computation state, the MoE execution engine reads the expert's weight data from the VRAM expert cache and completes the computation of the hybrid expert model to obtain the inference output.

[0121] On the other hand, the system continuously monitors the cache capacity status of VRAM and DRAM. When the available capacity of a certain storage tier falls below a preset capacity threshold, space needs to be freed up for that tier to accommodate higher-value expert weights. At this point, a degradation triggering process is initiated. After degradation is triggered, the system combines the priority scores of candidate experts and historical expert access sequences to predict the short-term reuse probability of each expert and performs a replacement selection accordingly. The replacement selection process determines which expert weights should be migrated from the current storage tier.

[0122] For selected experts, if their short-term reuse probability is less than or equal to the first reuse probability threshold, but there is still a possibility of near-term reuse, the system downgrades their weight from VRAM to DRAM, i.e., reverts from VRAM-resident state to DRAM temporary storage state. If their short-term reuse probability is further less than the second reuse probability threshold, indicating that the expert is unlikely to be reused in the short term, the system directly removes their weight from VRAM or DRAM to NAND. For experts with extremely low short-term reuse probability (below the third reuse probability threshold), the system performs compression or quantization on the expert's weight before removing them from NAND to save NAND storage space and reduce write wear. After the removal operation is completed, the expert's weight returns to the NAND-only state.

[0123] also, Figure 7 The document also illustrates the write-back or record update process and the metadata update process throughout all states. After each prefetch, promotion, demotion, or eviction, the system asynchronously persists the changed weight data or metadata to NAND via operation logs and a backfill queue, ensuring that NAND always retains a valid copy of the weights and the latest state information. Simultaneously, the system synchronously updates metadata fields such as the residency status, level priority, last access time, compression status, and NAND physical location of each expert, ensuring that every subsequent cache management decision is based on the latest system state, thereby guaranteeing the continuous and efficient operation of the multi-level cache management mechanism.

[0124] In one embodiment, one or more data tables may be maintained. These data tables may be implemented as in-memory tables, SSD-resident key-value records, metadata pages, log-structured metadata areas, or data structures shared between the host and the SSD.

[0125] The expert metadata table can use the expert identifier and tier identifier as primary keys. The tier mapping table can record the expert's current resident tier and the location of valid replicas. The operation log can record prefetch, promotion, demotion, eviction, backfilling, and failure recovery events. The physical layout index can map expert shards to NAND logical block address ranges, page groups, object keys, namespaces, or partition locations. It should be understood that this is only an example; actual implementations are not limited to this. For example, some fields can be merged, different tables can be split, or separate partial views can be maintained for the SSD controller, host runtime, and model execution engine.

[0126] In one embodiment, a system is also provided that includes a processor, memory, an SSD or AI SSD, an inference runtime, and an accelerator for executing hybrid expert models. The SSD or AI SSD includes NAND flash memory media, an SSD controller, firmware, a cache buffer, a metadata area, and an interface for reading expert weights.

[0127] In the host-side implementation, the inference runtime maintains expert metadata and caching strategies, and reads expert weights from the SSD through a regular storage interface. In the SSD-side or AI SSD implementation, some metadata queries, physical layout indexes, prefetch request sorting, compression or decompression, and backfill log updates can be performed by the SSD controller, embedded processor, firmware, or near-memory computing logic.

[0128] NAND media can store expert weight objects, compressed or quantized versions, metadata pages, mapping tables, operation logs, and backfill queues.

[0129] Accordingly, embodiments of this application also provide a weight processing device for an expert model of a discrete graphics card architecture. For example... Figure 8 As shown, the weight processing device 80 of the expert model of the discrete graphics card architecture includes a candidate expert determination unit 801, a metadata update unit 802, a cache management operation unit 803, and a model inference unit 804.

[0130] The candidate expert determination unit 801 is used to determine at least one candidate expert and the activation probability of each candidate expert based on the routing output during the inference process of the expert model. This unit may include, for example, a routing output analyzer or an activation predictor. By analyzing the routing output, the candidate expert determination unit filters out potentially activated candidate experts from all experts, providing target object and activation probability information for subsequent metadata updates and cache management operations.

[0131] The metadata update unit 802 is used to update the metadata of candidate experts based on their activation probabilities. This unit can be implemented, for example, by accessing an expert metadata table. The metadata update unit ensures that the metadata of each candidate expert is updated in a timely manner at each inference step, enabling the cache management operation unit to make decisions based on the latest metadata.

[0132] The cache management operation unit 803 performs cache management operations on the weights of candidate experts across storage locations within storage tiers comprised of VRAM, DRAM, and NAND, based on the candidate experts' metadata. This unit may include, for example, an expert cache manager and a transmission and decoding engine. The cache management operation unit drives the migration of expert weights across multiple storage tiers based on metadata, enabling the aggregation of high-value expert weights to high-speed tiers and the migration of low-value expert weights to high-capacity tiers.

[0133] The model inference unit 804 is used to execute the inference process of the expert model based on the candidate experts after the cache management operation. The model inference unit performs inference calculations after the cache management operation is completed, ensuring that the expert weights involved in the calculation are ready in VRAM, and avoiding delays and pauses caused by waiting for weight loading during the inference process.

[0134] According to the weight processing device for the expert model of the discrete graphics card architecture provided in this application, during the inference process of the expert model, at least one candidate expert and the activation probability of each candidate expert are determined based on the routing output. This allows for the prediction of the expert weights of candidate experts that may be used in the subsequent inference process. The metadata of the candidate experts is updated based on their activation probabilities, and at least the activation probability metadata of the candidate experts is maintained. The activation status of each candidate expert is updated in real time. The updated metadata is used to perform cache management operations on the storage locations of the candidate expert weights across the storage hierarchy composed of VRAM, DRAM, and NAND. This allows for the pre-reading of candidate experts with higher usage probabilities to the faster read levels before the inference process is executed. Therefore, when the inference process of the expert model is executed based on the candidate experts after cache management operations, the inference latency can be significantly reduced. Furthermore, this method retrieves expert weights across storage levels based on one or more factors, including the activation probability of the candidate weights, access frequency, recent access time, and level priority. This fully considers the possibility of using candidate weights, achieving accurate matching between candidate experts and the current inference process, avoiding a large number of unnecessary read operations, and reducing read overhead.

[0135] The apparatus provided in this embodiment belongs to the same concept as the method provided in the above embodiments of this application, and can execute the method provided in any of the above embodiments of this application, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the specific processing content of the method provided in the above embodiments of this application, and will not be repeated here.

[0136] It should be understood that the units in the above device can be implemented by a processor calling software. For example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files to implement the functions of some or all of the above units. All units in the above device can be implemented entirely by a processor calling software, entirely by hardware circuits, or partially by a processor calling software with the remaining parts implemented by hardware circuits.

[0137] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0138] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0139] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0140] Another embodiment of this application also proposes a weight processing electronic device for an expert model on a discrete graphics card architecture, see [link to relevant documentation]. Figure 9 As shown, the device includes: Memory 200 and processor 210; The memory 200 is connected to the processor 210 and is used to store programs; The processor 210 is configured to implement the weight processing method for expert models with discrete graphics card architecture disclosed in any of the above embodiments by running programs stored in the memory 200. The electronic device may be, for example, an edge device, a personal workstation, an embedded inference device, an AI PC, a local inference device, or a storage-enhanced inference server. The electronic device may also include a graphics processing unit (GPU) or other accelerator for providing VRAM and performing inference computations for the expert model. The electronic device may also include a solid-state drive (SSD) or AISSD for providing a NAND storage layer. With the above hardware configuration, the electronic device can perform cache management operations based on expert-level metadata across the storage layers composed of VRAM, DRAM, and NAND, reducing inference latency and supporting the deployment of larger-scale expert models.

[0141] Specifically, the weight processing device for the aforementioned discrete graphics card architecture expert model may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0142] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them: A bus can include a pathway for transmitting information between various components of a computer system.

[0143] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0144] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0145] The memory 200 stores a program for executing the technical solution of this application, and may also store an operating system and other critical business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0146] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0147] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0148] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0149] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the weight processing method of any discrete graphics card architecture expert model provided in the above embodiments of this application.

[0150] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the weight processing method for an independent graphics card architecture expert model according to various embodiments of this application as described in any of the foregoing embodiments of this specification.

[0151] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0152] Furthermore, embodiments of this application can also be storage media, such as SSD NAND media, disks, optical disks, flash memory, or other non-volatile storage media. The computer program includes program instructions and / or data structures for implementing the weight processing method for the discrete graphics card architecture expert model shown in the various embodiments above. Data structures may include, for example, expert metadata tables, hierarchy mapping tables, operation logs, physical layout indexes, and other data structures used to support multi-level cache management. By incorporating this storage media, the device can perform cache management operations based on expert-level metadata across storage tiers composed of VRAM, DRAM, and NAND, reducing inference latency and supporting the deployment of larger-scale expert models.

[0153] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0154] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are fundamentally similar to method embodiments, the descriptions are relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0155] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0156] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0157] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0158] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0159] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0160] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0161] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0162] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0163] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A weight processing method for an expert model of a discrete graphics card architecture, characterized in that, include: During the reasoning process of the expert model, at least one candidate expert and the activation probability of each candidate expert are determined based on the routing output. The metadata of the candidate expert is updated based on the activation probability of the candidate expert. The metadata includes the activation probability of the candidate expert, and also includes one or more of the following: access frequency, recent access time, and hierarchical priority. Based on the metadata of the candidate experts, cache management operations are performed on the storage locations of the candidate experts' weights across the storage hierarchy consisting of VRAM, DRAM, and NAND. The reasoning process of the expert model is executed based on the candidate experts after the cache management operation.

2. The method according to claim 1, characterized in that, Determining at least one candidate expert and the activation probability of each candidate expert based on the routing output includes: Based on the routing output, experts whose activation probability evaluated by the router is greater than a preset threshold are identified as candidate experts, and the activation probability of each candidate expert is determined.

3. The method according to claim 1, characterized in that, The metadata also includes access frequency, recent access time, hierarchy priority, current inference stage, and cache capacity status; performing cache management operations on the storage locations of the candidate experts' weights within the storage hierarchy consisting of VRAM, DRAM, and NAND, based on the candidate experts' metadata, includes: The priority score of each candidate expert is determined based on the candidate expert's metadata; Based on the priority scores of the candidate experts, cache management operations are performed on the storage locations of the candidate experts in the storage hierarchy consisting of VRAM, DRAM, and NAND. The cache management operations include one or more of the following: prefetching, promotion, demotion, eviction, and backfilling.

4. The method according to claim 3, characterized in that, Determining the priority score of the candidate experts based on their metadata includes: The candidate experts are processed using a preset scoring method based on one or more of the following factors: activation probability, access frequency, recent access time, hierarchical priority, current inference stage, and cache capacity status, to obtain a priority score for the candidate experts. The preset scoring methods include one or more of the following: rule-based scoring, weighted scoring, learning predictor, least frequently used algorithm, and least recently used algorithm.

5. The method according to claim 3, characterized in that, The cache management operations include prefetching and boosting. The cache management operations, performed on the weights of the candidate experts based on their priority scores at storage locations within the storage hierarchy comprised of VRAM, DRAM, and NAND, include: If the priority score of the candidate expert is greater than or equal to the first score threshold, the weight of the candidate expert is prefetched into DRAM or VRAM. or, If the priority score of the candidate expert is greater than or equal to the second score threshold and lower than the first score threshold, then the weight of the candidate expert is increased from NAND to VRAM. Among them, the first scoring threshold is greater than the second scoring threshold.

6. The method according to claim 5, characterized in that, The metadata also includes compression status; increasing the weight of the candidate expert from NAND to DRAM or VRAM includes: If the weights of the candidate experts are determined to be stored in NAND in compressed or quantized form based on the compression status in the metadata, then the weights of the candidate experts are read from NAND to DRAM, decompressed or dequantized, and then promoted to VRAM.

7. The method according to claim 3, characterized in that, The cache management operations include demotion and eviction. The cache management operations, performed based on the priority scores of the candidate experts and their weights at storage locations within the storage hierarchy comprised of VRAM, DRAM, and NAND, include: Based on the priority scores of the candidate experts and the historical expert access sequence, the short-term reuse probability of each candidate expert is predicted. If the short-term reuse probability of the candidate expert is less than or equal to the first reuse probability threshold, and the available capacity of the storage layer where the weight of the candidate expert is located is lower than the capacity threshold, based on the cache capacity status in the metadata of the candidate expert, then the weight of the candidate expert is downgraded from VRAM to DRAM, or from DRAM to NAND. or, If the short-term reuse probability of the candidate expert is less than the second reuse probability threshold, and the available capacity of the storage layer where the weight of the candidate expert is located is lower than the capacity threshold, based on the cache capacity status in the metadata of the candidate expert, then the weight of the candidate expert will be eliminated to NAND. Among them, the first reuse probability threshold is greater than the second reuse probability threshold.

8. The method of claim 7, wherein the metadata further includes a compressed state; and transferring the weights of the candidate experts from elimination to NAND includes: If the short-term reuse probability of the candidate expert is lower than the third reuse probability threshold, then the weight of the candidate expert is compressed and eliminated into NAND. Among them, the third reuse probability threshold is less than the second reuse probability threshold.

9. The method according to any one of claims 1-8, characterized in that, Also includes: After performing cache management operations, the metadata of each expert within the storage hierarchy is updated according to the cache management operations.

10. A weight processing device for an expert model of a discrete graphics card architecture, characterized in that, include: The candidate expert determination unit is used to determine at least one candidate expert and the activation probability of each candidate expert based on the routing output during the inference process of the expert model. The metadata update unit is used to update the metadata of the candidate expert according to the activation probability of the candidate expert. The metadata includes the activation probability of the candidate expert, and also includes one or more of the following: access frequency, recent access time, and hierarchical priority. A cache management operation unit is used to perform cache management operations on the storage locations of the candidate experts' weights in the storage hierarchy consisting of VRAM, DRAM, and NAND, based on the candidate experts' metadata. The model inference unit is used to execute the inference process of the expert model based on the candidate experts after the cache management operation.

11. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the weight processing method for the expert model of the discrete graphics card architecture as described in any one of claims 1 to 9 by running the program in the memory.

12. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the weight processing method for the expert model of the discrete graphics card architecture as described in any one of claims 1 to 9.

Images