Uma architecture expert model inference request processing method and apparatus, and related device

Abstract

The application provides a reasoning request processing method and device of a UMA architecture expert model, electronic equipment and a readable medium. The method comprises the following steps: obtaining a reasoning request of an expert model; determining the metadata of each expert in the expert model based on the history record of the routing selection of the weight of each expert in the expert model; determining the current token hidden state and the routing result of the current calculation window according to the reasoning request of the expert model; obtaining at least one candidate expert according to the current token hidden state, the routing result and the history expert calling sequence prediction; reading the expert weight from NAND to VRAM or DRAM according to the metadata of the candidate expert when performing the reasoning calculation of the current calculation window according to at least one active expert; and performing the next token reasoning calculation according to the read candidate expert in the next calculation window of the reasoning request. The technical scheme can significantly reduce the response delay of the reasoning process and improve the response speed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and readable medium for processing inference requests using a UMA architecture expert model. Background Technology

[0002] With the increasing demand for Mixture of Experts (MoE) models deployed on edge devices, personal workstations, and edge servers, there is a need to run large-scale models with massive parameters on heterogeneous computing platforms with a Unified Memory Architecture (UMA) at a lower hardware cost. However, the local high-speed memory (such as VRAM) of accelerators in such platforms is highly limited in capacity, often insufficient to support all expert weights of the model. Therefore, it is necessary to utilize larger-capacity but slower-access system memory (DRAM) or even NAND memory with unified addressing or page migration mechanisms to manage data flow across different storage tiers. For the inference process of MoE models, the corresponding expert weights are typically read only after the routing layer has fully determined the expert activated by the current input. When the required expert weights are located on NAND media, this process introduces a series of time-consuming operations that must be completed sequentially, including reading from the NAND media, possible decompression and recovery, and moving data via Direct Memory Access (DMA) to high-speed memory. This process introduces delays on the critical time path of token inference in the model, slowing down the overall response time.

[0003] In summary, reducing the response latency of expert models in a unified memory architecture is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] Based on the defects and shortcomings of the prior art, this application proposes a reasoning request processing method, apparatus, electronic device, and readable medium for an MA architecture expert model.

[0005] The first aspect of this application provides a method for processing inference requests of an expert model in a UMA architecture, comprising: obtaining an inference request of the expert model; determining the metadata of each expert in the expert model based on the historical records of the selection of each expert weight by routing; determining the current token hidden state and routing result of the current computation window according to the inference request of the expert model, wherein the routing result includes at least one active expert; predicting at least one candidate expert among the experts in the expert model according to the current token hidden state, routing result and historical expert call sequence; when performing inference computation of the current computation window according to at least one active expert, reading the expert weight of the candidate expert from NAND to VRAM or DRAM according to the metadata of at least one candidate expert; and performing inference computation of the next token according to the read candidate expert in the next computation window of the inference request.

[0006] In one embodiment, the metadata of each expert in the expert model is determined based on the historical records of each expert weight being selected by the route. This includes determining the metadata of each expert based on the hierarchical position, access popularity, historical route hit records, activation probability, and hierarchical memory pressure of each expert weight. The hierarchical position is the storage location of each expert weight in the storage hierarchy composed of VRAM, DRAM, and NAND. The routing results also include the activation probability corresponding to each activated expert. The access popularity is obtained based on the historical expert call sequence.

[0007] In one embodiment, access popularity includes hot state, warm state, and cold state; reading the expert weights of candidate experts from NAND to VRAM or DRAM based on the metadata of at least one candidate expert includes: reading the expert weights of candidate experts with hot access popularity and at least one active expert determined according to the routing results into VRAM; and / or, reading the expert weights of candidate experts with warm access popularity into DRAM; and / or, retaining the expert weights of candidate experts with cold access popularity in NAND.

[0008] In one embodiment, retaining candidate experts with a cold access status in the NAND includes: retaining the expert weights of candidate experts with a cold storage status in the NAND in a compressed or quantized form.

[0009] In one embodiment, the method further includes: determining a model partition for each expert weight in the expert model, the model partition including a common layer, an attention layer, a routing layer, and an expert layer; determining the storage location of each expert weight based on the model partition for each expert weight and one or more elements in the metadata of each expert; and storing each expert weight according to the storage location of each expert weight.

[0010] In one embodiment, predicting at least one candidate expert from among the experts in the expert model based on the current token hidden state, routing results, and historical expert call sequences includes: determining the semantic features of the current token based on the current token hidden state; and predicting at least one candidate expert based on the semantic features of the current token, the activation probability of at least one activating expert, and the historical expert call sequence.

[0011] In one embodiment, reading the expert weight of a candidate expert from NAND to VRAM or DRAM based on the metadata of at least one candidate expert includes: updating the metadata of the candidate expert based on the current token hidden state and routing results; and reading the expert weight of the candidate expert from NAND to VRAM or DRAM based on the updated metadata of the candidate expert.

[0012] A second aspect of this application provides an inference request processing apparatus for a UMA architecture expert model, comprising: a request acquisition unit for acquiring an inference request for the expert model; a metadata maintenance unit for determining the metadata of each expert in the expert model based on the historical records of the selection of each expert weight by routing in the expert model; a routing processing unit for determining the current token hidden state and routing result of the current computation window according to the inference request of the expert model, wherein the routing result includes at least one active expert; a candidate expert determination unit for predicting at least one candidate expert among the experts in the expert model according to the current token hidden state, the routing result, and the historical expert call sequence; a weight reading unit for reading the expert weight of the candidate expert from NAND to VRAM or DRAM according to the metadata of the at least one candidate expert when performing inference computation of the current computation window according to the at least one active expert; and an inference execution unit for performing inference computation of the next token according to the read candidate expert in the next computation window of the inference request.

[0013] A third aspect of this application provides an electronic device, including a memory and a processor; the memory is connected to the processor and is used to store a program; the processor is used to implement the inference request processing method of the UMA architecture expert model provided in any implementation of the first aspect by running the program in the memory.

[0014] The fourth aspect of this application provides a storage medium on which a computer program is stored. When the computer program is run by a processor, it implements the reasoning request processing method for the UMA architecture expert model provided in any of the implementations of the first aspect above.

[0015] The inference request processing method for the UMA architecture expert model proposed in this application determines the metadata of each expert based on the historical records of expert weights being selected by routing when processing inference requests. Then, within the current computation window, it predicts at least one candidate expert that may be used in future computation windows based on the current token hidden state, routing results, and historical expert call sequences. Furthermore, while executing the inference computation of the currently active expert, it asynchronously reads the expert weights of the candidate experts from NAND flash memory into VRAM or DRAM in parallel based on the candidate experts' metadata. In the next computation window, the pre-fetched candidate experts are directly used to perform inference computation. Thus, the latency of NAND reading, decompression recovery, and data transfer is hidden within the existing computation time of the current computation window. Expert computation for the next token does not need to wait for NAND loading, significantly reducing the response latency of the inference process and improving response speed. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the inference request processing method for the UMA architecture expert model provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of model partitioning and layering provided in an embodiment of this application.

[0019] Figure 3 This is a schematic diagram of the architecture of a UMA architecture hybrid expert model inference system provided in an embodiment of this application.

[0020] Figure 4 This is a schematic diagram of a runtime residency check and migration process provided in an embodiment of this application.

[0021] Figure 5 This is a schematic diagram illustrating the overlapping timing of computation and NAND access pipelines provided in an embodiment of this application.

[0022] Figure 6 This is a schematic diagram of the metadata feedback and policy adaptation process provided in one embodiment of this application.

[0023] Figure 7 A schematic diagram of an inference request processing device for a UMA architecture expert model provided in an embodiment of this application.

[0024] Figure 8This is a schematic diagram of an electronic device for inference request processing of a UMA architecture expert model according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] This application relates to a scheme for processing inference requests for expert models on a heterogeneous computing platform with a unified memory architecture. The Hybrid Expert Model (MoE) expands model capacity through multiple expert sub-networks, and during inference, a routing mechanism selects a subset of experts for each token to participate in computation. On the edge-side heterogeneous computing platform, the video random access memory (VRAM) has limited capacity, making it difficult to keep all expert weights permanently resident in the high-speed device memory. Current solutions read the required expert weights from low-speed storage media only after the routing result is completely determined, which exposes the read latency directly on the critical path of inference, causing inference blocking. How to reduce the impact of expert weight loading on inference latency under limited high-speed memory conditions is the technical problem that this application aims to solve.

[0027] Therefore, a new method, apparatus, electronic device, and readable medium for processing inference requests for the UMA architecture expert model are needed.

[0028] The basic concepts and overall process involved in the embodiments of this application are explained below.

[0029] UMA refers to a heterogeneous computing platform where DRAM on the Central Processing Unit (CPU) side, VRAM on the Graphics Processing Unit (GPU) or accelerator side, and NAND media on the storage side are uniformly addressed or managed at runtime. In other embodiments, the UMA architecture can also be a consistent memory pool architecture where the CPU and accelerator share system memory. The expert model in the embodiments of this application can be, for example, a MoE model. The MoE model expands its capacity through multiple expert sub-networks, and during inference, a routing layer or gating mechanism selects a subset of experts to participate in the computation for each token.

[0030] Figure 1 This is a flowchart illustrating a reasoning request processing method for a UMA architecture expert model provided in an embodiment of this application. Figure 1 As shown, the steps of this application include: Step S101: Obtain the inference request from the expert model.

[0031] The inference request could be, for example, a token-level inference task request for an expert model, which triggers the system to obtain the current token status, routing observation information, and runtime resource pressure information.

[0032] Step S102: Based on the historical records of the selection of routes by the weights of each expert in the expert model, determine the metadata of each expert in the expert model.

[0033] This metadata is used to record the storage status and access characteristics of each expert's weight. By determining the metadata based on historical routing selection records, the system can distinguish the access frequency and storage location of different experts, providing a data foundation for differentiated residency and migration strategies and avoiding the use of an indiscriminate storage management approach for all experts.

[0034] Step S103: Determine the current token hidden state and routing result of the current computation window based on the inference request of the expert model. The routing result includes at least one activated expert.

[0035] The current computation window here could be, for example, the inference computation stage corresponding to the current token or the current batch processing window. The current token hidden state refers to the intermediate representation of the current token obtained after processing by the model's pre-processing layers. This hidden state carries the semantic information of the current token and can be used for routing decisions and candidate expert prediction. The routing result is the expert selection result generated by the routing layer or gating network in the expert model based on the current token hidden state, including at least one activated expert selected to participate in the inference computation of the current computation window.

[0036] Step S104: Based on the current token hidden state, routing results, and historical expert call sequences, predict at least one candidate expert among the experts in the expert model.

[0037] Historical expert call sequences refer to the timeline of how experts were selected and actually participated in computation during previous inference processes. These sequences reflect the temporal patterns and contextual relationships of expert activation. By combining the current token hidden state and routing results with candidate expert prediction, the sequence dependencies of expert calls can be captured, improving prediction accuracy. This prediction is completed before or during the inference computation of the current computation window, thus providing a time window for preloading candidate expert weights. By predicting candidate experts for the next computation window within the current computation window, the system can advance NAND read operations to parallel execution within the current computation window's computation time, allowing storage access latency to overlap with computation time and preventing the next computation window from being blocked while waiting for expert weights to load.

[0038] Step S105: When performing inference calculations for the current calculation window based on at least one activated expert, the system reads the expert weights of the candidate experts from NAND to VRAM or DRAM based on the metadata of at least one candidate expert.

[0039] In other words, the inference calculation of the current calculation window and the loading operation of candidate expert weights overlap in time. For example... Figure 1 As shown, while the GPU or accelerator performs computations at the current layer, the system performs NAND reads, decompression and recovery, and Direct Memory Access (DMA) transfer operations for candidate experts in parallel. By pipelinedly overlapping the NAND reads of candidate expert weights with the inference computations of the current computation window, NAND access latency is hidden within the existing computation time, reducing the visible wait time on the critical path of inference.

[0040] The aforementioned reading of candidate expert weights from NAND to VRAM refers to loading the candidate expert weights into the high-speed device memory on the GPU or accelerator side, allowing subsequent inference computations to be executed directly on the accelerator. After the candidate expert weights reside in VRAM, the expert feedforward computation for the next computation window can be executed directly on the GPU or accelerator with minimal access latency, without additional data transfer steps. Reading candidate expert weights from NAND to DRAM refers to loading the candidate expert weights into the CPU-side host memory for temporary storage. In some embodiments, the weights temporarily stored in DRAM can be further increased to VRAM via DMA as needed. When the VRAM capacity is insufficient to accommodate all candidate expert weights, some candidate expert weights are first loaded into the DRAM layer for temporary storage, avoiding the inability to load candidate experts in advance due to insufficient VRAM space, while retaining flexibility for subsequent on-demand increases to VRAM.

[0041] Step S106: In the next calculation window of the inference request, perform the inference calculation of the next token based on the candidate experts that have been read.

[0042] Once the routing result for the next computation window is determined, if the weights of the required experts have already been loaded into VRAM or DRAM as candidate experts in the previous prefetch operation, the system can directly use the resident expert weights to perform expert feedforward computation without having to read from NAND again. Since the candidate expert weights have been preloaded within the computation time of the next computation window (i.e., the current computation window mentioned above), the inference computation of the next computation window can immediately obtain the required weights and execute, eliminating inference blocking caused by waiting for NAND reads, reducing response latency in the inference process, and improving overall response speed.

[0043] According to the inference request processing method for the UMA architecture expert model provided in this application, when processing inference requests, the method determines the metadata of each expert based on the historical records of expert weights being selected by routing. In the current computation window, it accurately predicts at least one candidate expert that may be used in future computation windows based on the current token hidden state, routing results, and historical expert call sequences. Then, when performing inference computation for the currently active expert, it asynchronously reads the expert weights of the candidate experts from NAND flash memory to VRAM or DRAM in parallel based on the candidate experts' metadata. In the next computation window, the pre-fetched candidate experts are directly used to perform inference computation. Thus, the latency of NAND reading, decompression recovery, and transfer is hidden within the existing computation time of the current computation window. Expert computation for the next token does not need to wait for NAND loading, significantly reducing the response latency of the inference process and improving response speed.

[0044] In some embodiments, Figure 1 Step S102, as shown, can further determine the metadata of each expert based on the hierarchical position, access frequency, historical route hit records, activation probability, and hierarchical memory pressure of each expert's weight. By determining the metadata of each expert through multi-dimensional information such as hierarchical position, access frequency, historical route hit records, activation probability, and hierarchical memory pressure, the system can comprehensively consider the storage location of expert weights, access frequency, probability of being selected by a route, and the current scarcity of storage resources, thereby improving the prediction accuracy of candidate experts in the subsequent step S104. Table 1 exemplarily illustrates the metadata fields.

[0045] Table 1 Representative metadata fields The tier position refers to the current storage location of each expert's weight within the storage hierarchy composed of VRAM, DRAM, and NAND. VRAM is the high-speed device memory layer (hot layer); DRAM is the host memory layer (warm layer); and NAND is the high-capacity storage medium layer (cold layer). The read speed comparison of the three storage layers is: VRAM > DRAM > NAND. Table 2 provides an example of the storage tiers and their residency functions.

[0046] Table 2 Storage Hierarchy and Residency Function Access popularity is obtained based on historical expert call sequences and is used to characterize how frequently each expert is selected in recent inference processes. Access popularity based on historical expert call sequences can reflect the actual usage frequency of each expert.

[0047] Historical route hit records are used to document how each expert was selected by the routing layer during the historical inference process, which can assist in prefetching and elimination decisions. Historical route hit records provide detailed time-series and frequency information on the selection of each expert, allowing the system to identify expert activation patterns and trends, and providing historical evidence for predicting candidate experts.

[0048] The routing results also include the activation probability of each activated expert. This activation probability is output by the routing layer or gating network when generating the routing results, and represents the confidence or weight allocation of each expert selected to participate in the current calculation.

[0049] Tiered memory pressure is used to characterize the current capacity usage of high-speed storage tiers such as VRAM and DRAM. For example, tiered memory pressure can be determined based on runtime monitoring metrics such as VRAM free capacity and DRAM buffer usage. Incorporating tiered memory pressure into metadata allows the system to be aware of the current storage resource strain when making migration decisions.

[0050] The access heat mentioned above can be further subdivided into different levels to achieve more granular hierarchical management. In some embodiments, access heat includes hot, warm, and cold states. In step S106, when reading the expert weights of candidate experts from NAND to VRAM or DRAM based on the metadata of at least one candidate expert, the expert weights of candidate experts with a hot access heat state, as well as at least one active expert determined according to the routing results, can be read into VRAM; and / or, the expert weights of candidate experts with a warm access heat state can be read into DRAM; and / or, the expert weights of candidate experts with a cold access heat state can be retained in NAND. The above method can prioritize fast access to high-frequency experts within limited high-speed memory capacity, while reducing unnecessary NAND access and repeated data transfer.

[0051] Reading the expert weights of candidate experts with a "hot" access status into VRAM refers to loading the weights of candidate experts who have been frequently selected by routing recently into the high-speed device memory on the GPU or accelerator side. Hot-status candidate experts have a higher probability of being reactivated in subsequent computation windows; residing their weights in VRAM ensures that subsequent inference computations are executed directly on the accelerator with minimal latency, avoiding repeated loading from low-speed levels. Simultaneously, the expert weights of at least one activated expert determined based on the routing results are also read into VRAM to execute the current inference computation. Activated experts are direct participants in the inference computation of the current computation window; their weights must reside in VRAM for the GPU or accelerator to perform expert feedforward computations. Reading them into VRAM along with hot-status candidate experts ensures the normal execution of the current inference computation.

[0052] Loading the expert weights of candidate experts in a "warm" access state into DRAM refers to temporarily storing the weights of candidate experts who may be accessed recently but whose access frequency is less than that of "warm" experts in the CPU-side host memory. Warm-state candidate experts have a certain probability of being activated, but not enough to occupy the limited VRAM capacity. Preloading their weights into DRAM ensures that when the expert is actually selected in a subsequent calculation window, only a DRAM-to-VRAM DMA transfer is needed to complete the promotion, avoiding the longer latency of rereading from NAND.

[0053] Retaining the expert weights of candidate experts in a "cold" state (low access frequency) in NAND flash memory means that for low-frequency candidate experts who have not been frequently selected in routing over a long period, their weights are not actively loaded into the high-speed storage layer, but instead remain in the NAND medium. Cold-state candidate experts have a low probability of being activated in the near future; retaining their weights in NAND flash memory avoids occupying limited VRAM and DRAM capacity, allowing high-speed storage resources to be reserved for more likely-to-be-used "hot" and "warm" state experts, thereby improving the utilization efficiency of high-speed memory.

[0054] The "AND / OR" relationship between the three tiered placement operations mentioned above indicates that these operations can be executed individually or in any combination. For example, the system can execute only the operation of reading hot-state candidate experts into VRAM; it can also execute the operations of reading hot-state candidate experts into VRAM and warm-state candidate experts into DRAM simultaneously; or it can execute all three operations simultaneously. Through this flexible combination, the system can dynamically determine which one or more tiered placement operations to execute based on the current memory pressure, the number of candidate experts, and the distribution of access popularity, achieving reasonable allocation of storage resources under different operating conditions.

[0055] Furthermore, in some embodiments, the expert weights of candidate experts in a cold storage state are retained in the NAND flash memory in a compressed or quantized form. This reduces the storage volume of the expert weights, thereby reducing the NAND flash memory footprint and minimizing the amount of data read from the NAND flash memory when backfilling is required, thus shortening the read time. The compression method can be, for example, block compression, sparse coding, differential coding, or a combination thereof, which reduces storage volume by eliminating redundant information in the weight data. Storing cold-state candidate expert weights in a compressed form allows for more expert weights to be accommodated in scenarios with limited NAND flash memory capacity, while also reducing the amount of raw data read from the NAND flash memory when backfilling is required. However, a decompression and recovery operation needs to be performed before use, and this recovery operation can also be pipelined and overlapped with other computational steps (see...). Figure 5(Example) to reduce the impact on inference latency. Quantization can be, for example, converting expert weights from a high-precision floating-point representation to a low-precision fixed-point representation (e.g., quantizing from FP16 to INT8 or INT4). Quantization can reduce the storage bit width of each weight parameter to reduce storage volume, and the processing overhead of restoring the quantized weights to computational precision is generally lower than the decompression overhead of general compression algorithms, which helps to shorten the recovery time on the backfill path.

[0056] The tiered placement strategy based on access popularity has been explained above. The following section will further explain the differentiated storage management based on model partitioning.

[0057] In some embodiments, a model partition for each expert weight in the expert model may be determined, the model partition including a common layer, an attention layer, a routing layer and an expert layer; the storage location of each expert weight may be determined based on the model partition for each expert weight and one or more elements in the metadata of each expert; and each expert weight may be stored according to the storage location of each expert weight.

[0058] Different model partitions play different functional roles, have different access frequencies and importance during inference. Differentiated storage management by partition can ensure that the weights of critical paths reside preferentially in high-speed tiers, while the weights of non-critical paths are reasonably offloaded to low-speed tiers, thereby achieving better storage resource allocation within limited high-speed memory capacity. Table 3 shows an example of model layering and its purpose.

[0059] Table 3 Model layering and purpose of layering The common layer can include basic model components used by all token inference, such as word embedding layers, normalization layers, and shared projection layers. The weights of the common layer are accessed during the inference process of each token and are fundamental components on the critical inference path. By identifying the common layer as an independent partition, the system can prioritize ensuring it resides in a high-speed layer (such as VRAM or DRAM), avoiding the impact of storage access latency on the inference of each token due to the unloading of the common layer weights.

[0060] The attention layer may include, for example, attention weights and projection matrices, intensive computational components used to support the current token or computation window. Attention layer weights are frequently accessed during intensive token processing in the current computation window. After identifying them as independent partitions, the system can manage their residency state at the granularity of layer window switching. Attention layer weights are kept in VRAM during the currently active window, and attention layer weights of inactive windows can be backed up to DRAM during window switching.

[0061] The routing layer can include model components such as expert selection networks or gating networks to generate expert call results. The routing layer is responsible for generating routing results for each token to determine the activating expert, and is a crucial step in expert selection. By recognizing the routing layer as an independent partition, the system can set it to reside permanently in VRAM to avoid routing decision delays caused by the unloading of routing layer weights, ensuring the real-time nature of the expert selection process.

[0062] The expert layer, for example, may include the weights of each expert's feedforward network, and is the main control object for differentiated management based on activation popularity in the MoE model. The expert layer contains the weights of numerous expert sub-networks, representing the largest storage footprint in the MoE model, and is also the main control object for hierarchical placement and unloading inference in this embodiment. After identifying the expert layer as an independent partition, the system can perform fine-grained hot, warm, and cold classification and differentiated storage management of the expert weights within the expert layer based on metadata such as the access popularity and activation probability of each expert.

[0063] Figure 2 This diagram illustrates model partitioning and hierarchical placement according to an embodiment of this application. The storage operation can be completed collaboratively by a unified address mapping manager and an asynchronous migration engine. After storage, the corresponding unified address mapping entries are updated. Table 4 exemplarily shows the unified address mapping entries. The actual weight placement operation is performed according to the determined storage location, enabling the model partitioning and metadata-driven hierarchical strategy to be implemented. Each expert weight is placed in a storage level that matches its access characteristics and functional role, laying the physical foundation for fast access and on-demand migration in the subsequent inference process.

[0064] Table 4 Uniform Address Mapping Entries The above has explained how metadata is determined and the storage management strategy based on model partitioning. The following section will further explain the detailed process of candidate expert prediction.

[0065] In some embodiments, Figure 1 The step S104 shown may further include S1041 to S1042.

[0066] Step S1041: Determine the semantic features of the current token based on the current token's hidden state.

[0067] For example, feature information reflecting the semantic direction of the current inference context can be extracted from the intermediate representation obtained after the current token has been processed by the model's pre-processing layers. Semantic features carry the semantic representation information of the current token in the model's intermediate layers, reflecting the topic and direction of the current inference context. Using semantic features as input for candidate expert prediction enables the prediction process to perceive the semantic content of the current token, thereby predicting the likelihood of experts related to the current semantic direction being activated in subsequent computation windows.

[0068] Step S1042: Based on the semantic features of the current token, the activation probability of at least one activation expert, and the historical expert call sequence, at least one candidate expert is predicted.

[0069] This prediction process comprehensively utilizes three types of information: semantic features provide the semantic direction of the current inference context, activation probabilities provide the confidence distribution of each expert in the current routing results, and historical expert call sequences provide the temporal patterns of expert activation. These three types of information characterize the probability of expert activation from different dimensions—semantic features reflect content-driven expert correlation, activation probabilities reflect the immediate tendency of the routing layer (experts close to the selection threshold have a higher probability of being activated in subsequent windows), and historical call sequences reflect the sequential dependencies of expert activation. Combining these three types of information improves the accuracy and robustness of candidate expert predictions and reduces invalid or missed prefetchings caused by prediction biases from single signals.

[0070] In the process of reading candidate expert weights from NAND to VRAM or DRAM, as mentioned above, in some embodiments, the metadata of the candidate experts can be updated based on the current token hidden state and routing results before performing the read operation, so that subsequent read and storage operations are performed based on the latest runtime information. By dynamically updating the metadata in each calculation window, changes in expert weights can be updated in a timely manner, avoiding inaccurate storage and read decisions based on the original metadata.

[0071] The metadata update process refers to the system updating the metadata fields (such as access popularity, historical route hit records, activation probability, hierarchical memory pressure, etc.) of candidate experts in each calculation window by using the semantic information carried by the current token hidden state and the activation experts and activation probability information in the routing results.

[0072] Figure 3This is a schematic diagram of the architecture of a hybrid expert model inference system based on the UMA architecture provided in one embodiment of this application. The computing platform on which this implementation environment is based is a unified memory heterogeneous computing platform, which integrates a central processing unit (CPU), a graphics processing unit (GPU) or other accelerators, system memory (DRAM), accelerator local high-speed memory (VRAM), and NAND flash memory storage media. At runtime, this platform provides a unified addressing or unified management view to the upper layers, but the weight data can still be physically distributed across different storage levels.

[0073] like Figure 3 As shown, the system includes an input stream module, a hybrid expert MoE inference runtime module, a model partition management module, a unified address mapping module, a route observation module, a residency policy engine module, an asynchronous migration engine module, a unified storage hierarchy module, a cache commit module, an expert computation module, and a layer output module. Arrows indicate the direction of data or control information transfer between modules.

[0074] The input stream module provides the input token sequence to the MoE inference runtime module. The MoE inference runtime module is responsible for layer scheduling during inference, transferring information about layers to be processed to the model partition management module. The model partition management module divides the weights in the hybrid expert model into model partitions such as common layers, attention layers, routing layers, and expert layers according to their structural roles, and provides logical identifiers related to these partitions to the unified address mapping module. The unified address mapping module maintains the mapping relationship between each model partition or expert weight and one or more physical storage locations, and records metadata information such as the current resident level, compression status, validity status, and migration status. The unified storage tier module abstractly manages three different physical storage tiers: the VRAM hot tier, the DRAM warm tier, and the NAND cold tier. The VRAM hot tier has the lowest access latency and lowest capacity, the DRAM warm tier has medium access latency and relatively large capacity, and the NAND cold tier has the largest storage capacity and relatively high access latency.

[0075] The routing observation module receives the current token hidden state from the inference process and generates a routing result, which includes at least the identifiers of one or more experts activated by the current token. The output of the routing observation module is connected to the residency policy engine module. The residency policy engine module also receives historical expert call sequences and runtime information reflecting current memory pressure, and generates residency, promotion, demotion, and migration decisions for expert weights based on these inputs, outputting the decisions to the asynchronous migration engine module and the cache commit module. The asynchronous migration engine module generates NAND read tasks, decompression and recovery tasks, and DMA transfer tasks to DRAM or VRAM for expert weights based on the decisions. The asynchronous migration engine module connects to the unified storage tier module and the expert computation module to complete the transfer of weight data between storage tiers and provide weights to the computation unit. The cache commit module is responsible for updating the residency state in the unified address mapping after the weights are successfully placed in the target tier and pass validity verification, so that the latest valid weight copy is visible during inference runtime. The expert computation module selects experts based on the routing results, performs feedforward network computation using the expert weights loaded into VRAM or DRAM, and outputs the layer output after fusing the results of each expert through the layer output module.

[0076] Figure 4 This is a schematic diagram illustrating a runtime residency check and migration process provided in one embodiment of this application. Figure 4 As shown, routing evaluation and expert set selection can be performed based on the current token state, and a residency check is implemented. The residency check is used to determine whether the corresponding expert weight is located in the VRAM hot layer, DRAM warm layer, or NAND cold layer. If the VRAM is hit, the expert feedforward calculation is directly performed; if not, the expert weight is backfilled to the target DRAM or VRAM through NAND batch reading, decompression recovery, and DMA transfer in sequence, and then submitted after caching for use in calculation. The process also includes result recording, backfilling decision, degradation / elimination, stability protection, and policy recording, forming a closed-loop control for runtime migration and scheduling. Table 5 shows an example runtime operation and triggering conditions.

[0077] Table 5 Runtime Operations and Triggering Conditions Based on the configuration of the residency check module, the system can comprehensively consider routing output, expert call history, current layer position, current token status, and layer resource pressure to prefetch candidate expert weights before demand arrives. When high-speed layer memory is insufficient or expert popularity decreases, offloading and degradation are triggered. For temporarily selected but not yet resided experts, a backfilling strategy is used to complete on-demand loading. The system also distinguishes between current window weights, next window weights, and background candidate weights. Current window weights have the highest scheduling priority, next window weights can be pre-read when NAND queues and DMA channels allow, and background candidate weights are warmed up or rearranged during low-load periods. Table 6 shows an exemplary prefetch, offload, and backfilling strategy.

[0078] Table 6 Prefetching, Unloading, and Backfilling Strategies In addition to the fields listed in Tables 3 and 4, the system can also maintain pipeline status (see...). Figure 5 Examples), rollback status, and operational metrics. These statuses and metrics can be recorded according to the data structures illustrated in Tables 7 to 9 to indicate the types of information that can be maintained by the host software, device firmware, or host-SSD co-architecture under different implementations, without limiting the specific field format.

[0079] Table 7. Flow line stages and overlap opportunities Table 8 Rollback and Consistency Handling Scenarios Table 9 Monitoring Indicators and Strategy Applications Figure 5 This is a schematic diagram illustrating the overlapping timing of computation and NAND access pipelines provided in an embodiment of this application.

[0080] like Figure 5 As shown, while the GPU performs computation at the current layer, the system simultaneously performs route observation, dwell decision-making, and reading and restoring of candidate expert weights. Specifically, during the computation at layer k, the system can perform route observation based on the current token state; during the computation at layer k+1, it performs dwell decision-making and reads the predicted experts; in subsequent stages, it further reads the candidate experts and sequentially performs weight restoration, rollback restoration, and copying operations to DRAM or VRAM, finally submitting the dwell table.

[0081] In one embodiment, by overlapping the GPU computation processes of layers k, k+1, and k+2 with operations such as NAND reads, decompression and recovery, DMA migration, and cache commits in time, the system can prefetch and place the expert weights required for the next computation window within the current computation window. Thus, the NAND read and recovery latency is hidden within the preceding computation process, thereby reducing the waiting time for subsequent expert computations and improving the efficiency of the inference pipeline.

[0082] Figure 6 This is a schematic diagram of the metadata feedback and policy adaptation process provided in one embodiment of this application.

[0083] like Figure 6 As shown, the system can perform policy evaluation based on operational metrics and expert metadata, and generate corresponding migration plans. Operational metrics include, for example, memory pressure, and expert metadata includes, for example, popularity scores. The policy evaluator can determine residency priorities based on this information and generate prefetch or unfetch migration plans accordingly.

[0084] The metadata update process is part of the metadata feedback control loop. Runtime metrics and expert metadata enter the policy evaluator, which generates a migration plan. After the actions are executed, measurement results are generated and written back to the policy state database and stability monitoring module. By updating metadata in each computation window based on the latest token hidden state and routing results, the system forms a runtime adaptive closed-loop feedback mechanism. This mechanism enables metadata to continuously track dynamic changes in expert access patterns, ensuring that subsequent prefetching, unloading, and backfilling decisions are executed based on the latest runtime state, achieving adaptive policy adjustment without retraining the MoE model.

[0085] After the metadata update is complete, the system reads the expert weights of the candidate experts from NAND to VRAM or DRAM based on the updated metadata. In other words, the system determines whether the candidate expert weights should be loaded into VRAM or DRAM, and the loading priority, based on information such as the latest access popularity, hierarchy position, and hierarchy memory pressure reflected in the updated metadata. Performing read decisions based on the updated metadata allows the system to select the most appropriate target storage hierarchy and loading priority according to the latest runtime conditions. For example, if the updated metadata shows that the access popularity of a candidate expert has increased from "warm" to "hot," the system can directly read its weight into VRAM instead of DRAM; if the updated metadata shows that VRAM memory pressure is high, the system can first read the candidate expert weights into DRAM to avoid crowding out high-priority expert weights already residing in VRAM.

[0086] In one embodiment, the system can perform corresponding actions according to the migration plan, such as performing a NAND read. After execution, the system obtains the corresponding measurement results, such as the hit rate, and writes the measurement results into the policy state database. The policy state database can maintain policy parameters such as threshold sets, and the stability monitoring module can implement oscillation protection based on the policy state and feed back relevant operating indicators to the subsequent policy evaluation process.

[0087] Thus, the system forms a closed-loop adjustment mechanism consisting of operational metrics, expert metadata, strategy evaluation, migration execution, result measurement, and stability monitoring, enabling prefetching, unloading, and dwell decisions to be adaptively adjusted according to runtime status.

[0088] The inference request processing method of the embodiments of this application has been described in detail above. The inference request processing device corresponding to the method will be described below.

[0089] Figure 7 This is a schematic diagram of an inference request processing device for a UMA architecture expert model provided in one embodiment of this application. Figure 7 As shown, the inference request processing device 70 of the UMA architecture expert model includes a request acquisition unit 701, a metadata maintenance unit 702, a routing processing unit 703, a candidate expert determination unit 704, a weight reading unit 705, and an inference execution unit 706.

[0090] The request retrieval unit 701 is used to retrieve the inference request of the expert model.

[0091] The metadata maintenance unit 702 is used to determine the metadata of each expert in the expert model based on the historical records of the selection of routes by the weights of each expert in the expert model.

[0092] The routing processing unit 703 is used to determine the current token hidden state and routing result of the current computation window based on the inference request of the expert model. The routing result includes at least one activated expert.

[0093] The candidate expert determination unit 704 is used to predict at least one candidate expert among the experts in the expert model based on the current token hidden state, routing results and historical expert call sequences.

[0094] The weight reading unit 705 is used to read the expert weights of at least one candidate expert from NAND to VRAM or DRAM based on the metadata of at least one candidate expert when performing inference calculations for the current calculation window based on at least one activated expert.

[0095] The inference execution unit 706 is used to perform inference computation of the next token based on the candidate experts that have been read in the next computation window of the inference request.

[0096] According to the inference request processing apparatus of the UMA architecture expert model of this application, when processing inference requests, the metadata of each expert is determined based on the historical records of expert weights being selected by routing. In the current computation window, at least one candidate expert that may be used in future computation windows is predicted based on the current token hidden state, routing results, and historical expert call sequences. Then, when performing inference computation for the currently active expert, the expert weights of the candidate experts are asynchronously read from NAND to VRAM or DRAM in parallel based on the candidate experts' metadata. In the next computation window, the pre-fetched candidate experts are directly used to perform inference computation. Thus, the latency of NAND reading, decompression recovery, and transfer is hidden within the existing computation time of the current computation window. Expert computation for the next token does not need to wait for NAND loading, significantly reducing the response latency of the inference process and improving response speed.

[0097] The apparatus provided in this embodiment belongs to the same concept as the method provided in the above embodiments of this application, and can execute the method provided in any of the above embodiments of this application, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the specific processing content of the method provided in the above embodiments of this application, and will not be repeated here.

[0098] It should be understood that the units in the above device can be implemented by a processor calling software. For example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files to implement the functions of some or all of the above units. All units in the above device can be implemented entirely by a processor calling software, entirely by hardware circuits, or partially by a processor calling software with the remaining parts implemented by hardware circuits.

[0099] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0100] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0101] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0102] Another embodiment of this application also proposes an inference request processing device for a UMA architecture expert model, see [link to relevant documentation]. Figure 8 As shown, the device includes: Memory 200 and processor 210; The memory 200 is connected to the processor 210 and is used to store programs; The processor 210 is configured to implement the inference request processing method for the UMA architecture expert model disclosed in any of the above embodiments by running the program stored in the memory 200.

[0103] Specifically, the inference request processing device of the above-mentioned UMA architecture expert model may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0104] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them: A bus can include a pathway for transmitting information between various components of a computer system.

[0105] Processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0106] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0107] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0108] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0109] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0110] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0111] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of the reasoning request processing method of any of the UMA architecture expert models provided in the above embodiments of this application.

[0112] This application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the inference request processing method for the UMA architecture expert model described in any of the above embodiments. For the specific processing procedure and its beneficial effects, please refer to the above-described embodiments of the inference request processing method for the UMA architecture expert model.

[0113] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the inference request processing method for the UMA architecture expert model according to various embodiments of this application as described in any of the above embodiments of this specification.

[0114] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0115] Furthermore, embodiments of this application may also be storage media storing a computer program. When the computer program is run by a processor, the processor executes the steps in the inference request processing method of the UMA architecture expert model according to various embodiments of this application described in any of the above embodiments of this specification. Specifically, the following steps can be implemented: Step S1101: Obtain the inference request from the expert model.

[0116] Step S1102: Based on the historical records of the selection of routes by the weights of each expert in the expert model, determine the metadata of each expert in the expert model.

[0117] Step S1103: Determine the current token hidden state and routing result of the current computation window based on the inference request of the expert model. The routing result includes at least one activated expert.

[0118] Step S1104: Based on the current token hidden state, routing results, and historical expert call sequences, predict at least one candidate expert among the experts in the expert model.

[0119] Step S1105: When performing inference calculations for the current calculation window based on at least one activated expert, the expert weights of the candidate experts are read from NAND to VRAM or DRAM based on the metadata of at least one candidate expert.

[0120] Step S1106: In the next calculation window of the inference request, perform inference calculation for the next token based on the candidate experts that have been read.

[0121] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0122] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are fundamentally similar to method embodiments, the descriptions are relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0123] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0124] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0125] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0126] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0127] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0128] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0130] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0131] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing inference requests in a UMA architecture expert model, characterized in that, include: Obtain the inference request from the expert model; Based on the historical records of the selection of routes by the weights of each expert in the expert model, the metadata of each expert in the expert model is determined. The current token hidden state and routing result of the current computation window are determined based on the inference request of the expert model, wherein the routing result includes at least one activated expert; Based on the current hidden state of the token, the routing result, and the historical expert call sequence, at least one candidate expert is predicted from among the experts in the expert model. When performing inference calculations for the current calculation window based on the at least one activated expert, the expert weights of the candidate experts are read from NAND to VRAM or DRAM based on the metadata of the at least one candidate expert. In the next calculation window of the inference request, the inference calculation for the next token is performed based on the candidate experts that have been read.

2. The method according to claim 1, characterized in that, Based on the historical records of each expert's weight being selected in the routing process within the expert model, the metadata of each expert in the expert model is determined to include: The metadata of each expert is determined based on the hierarchical position of each expert's weight, access popularity, historical route hit records, activation probability, and hierarchical memory pressure. Wherein, the hierarchical position is the storage position of each expert weight in the storage hierarchy composed of VRAM, DRAM and NAND; The routing results also include the activation probability corresponding to each activation expert; The access popularity is obtained based on the historical expert call sequence.

3. The method according to claim 2, characterized in that, The access popularity includes hot status, warm status and cold status; Reading the expert weights of the candidate experts from NAND to VRAM or DRAM based on the metadata of the at least one candidate expert includes: The candidate experts whose access popularity is hot, and the expert weights of at least one active expert determined according to the routing results, are read into VRAM. And / or, Read the expert weights of candidate experts whose access popularity is in the "warm" state into DRAM; And / or, The expert weights of candidate experts whose access popularity is low are retained in NAND.

4. The method according to claim 3, characterized in that, Retaining candidate experts with low access activity in NAND includes: The expert weights of candidate experts whose storage status is cold are retained in the NAND in a compressed or quantized form.

5. The method according to claim 2, characterized in that, Also includes: The model partitioning for each expert weight in the expert model is determined, and the model partitioning includes a common layer, an attention layer, a routing layer, and an expert layer; The storage location of each expert weight is determined based on the model partition of each expert weight and one or more elements in the metadata of each expert. The expert weights are stored according to their respective storage locations.

6. The method according to claim 2, characterized in that, Based on the current hidden state of the token, the routing result, and the historical expert call sequence, at least one candidate expert is predicted from among the experts in the expert model, including: The semantic features of the current token are determined based on the current token's hidden state; The at least one candidate expert is predicted based on the semantic features of the current token, the activation probability of the at least one activation expert, and the historical expert call sequence.

7. The method according to claim 1, characterized in that, Reading the expert weights of the candidate experts from NAND to VRAM or DRAM based on the metadata of the at least one candidate expert includes: Update the metadata of the candidate experts based on the current hidden state of the token and the routing results; The expert weights of the candidate experts are read from NAND to VRAM or DRAM based on the updated metadata of the candidate experts.

8. A reasoning request processing device for a UMA architecture expert model, characterized in that, include: The request retrieval unit is used to retrieve inference requests from the expert model; The metadata maintenance unit is used to determine the metadata of each expert in the expert model based on the historical records of the selection of each expert's weight by the route in the expert model. The routing processing unit is used to determine the current token hidden state and routing result of the current computing window based on the inference request of the expert model, wherein the routing result includes at least one activated expert; The candidate expert determination unit is used to predict at least one candidate expert from among the experts in the expert model based on the current hidden state of the token, the routing result, and the historical expert call sequence. The weight reading unit is used to read the expert weights of the candidate experts from NAND to VRAM or DRAM based on the metadata of the at least one candidate expert when performing inference calculations for the current calculation window based on the at least one activated expert. The inference execution unit is configured to perform inference calculations for the next token based on the read candidate experts in the next calculation window of the inference request.

9. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the inference request processing method for the UMA architecture expert model as described in any one of claims 1 to 7 by running the program in the memory.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the inference request processing method for the UMA architecture expert model as described in any one of claims 1 to 7.

Images