Memory access methods and inference systems based on CXL pooled memory and optical interconnects

By using an inference system based on CXL pooled memory and optical interconnects, and leveraging an early prediction network and a global shared memory pool, multiple constraints on memory access and computational resource management in the MoE model are resolved. This achieves optimization with large capacity, low latency, high bandwidth, and high computing power utilization, and supports elastic scaling.

CN123114794APending Publication Date: 2026-11-17SHENZHEN JINGTIE COMPUTING SYSTEM TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611279202.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies cannot simultaneously achieve high capacity, low latency, high bandwidth, high computing power utilization, and elastic expansion within the same architecture. In particular, the memory access and computing resource management of the MoE model suffer from problems such as idle memory, wasted bandwidth, cache pollution, and bundled computing resources.

Method used

An inference system based on CXL pooled memory and optical interconnect is adopted. Through lightweight UMA computing nodes and CXL optoelectronic fusion switching network, the early prediction network is used to predict the activation probability of experts, and dynamic prefetching and cache management are performed. Combined with global shared memory pool and optical interconnect technology, asynchronous prefetching and synchronous reading of expert weights are realized, thus optimizing memory access and calculation process.

Benefits of technology

It reduces the total cost of ownership of memory replicas and computing resources, reduces transmission latency, improves computing power utilization and memory capacity, supports elastic scaling, reduces the risk of interruption, and optimizes memory access efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114794A_ABST
    Figure CN123114794A_ABST
Patent Text Reader

Abstract

This invention discloses a memory access method and an inference system based on CXL pooled memory and optical interconnects, relating to the fields of computer architecture, storage access, and optical interconnects. The method includes: predicting the activation probability of each expert during the current Transformer layer attention calculation; determining a dynamic prefetch threshold based on the available bandwidth of the currently available CXL optical links, including experts with activation probabilities greater than the dynamic prefetch threshold in a candidate prefetch set, and sending asynchronous prefetch operations for the weight parameters of each expert in the candidate prefetch set to the global shared memory pool; after the current Transformer layer attention calculation is completed, if there are target expert weights that were not prefetched, initiating a synchronous read of the target expert weights; after the current Transformer layer inference is completed, reclaiming local cache space based on the heat value of each expert. This invention expands the available capacity of model weights through pooled memory and reduces the impact of remote expert weight access on the critical path of inference through asynchronous prefetching based on expert activation probabilities and link states.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of computer architecture, storage access, and optical interconnect technology, specifically to a memory access method and an inference system based on CXL pooled memory and optical interconnect. Background Technology

[0002] As the parameter scale of Large Language Models (LLMs) continues to evolve from hundreds of billions to trillions, the Mixture of Experts (MoE) architecture, with its characteristics of "large total parameter scale and sparse activation in single inference," has become an important technical approach to improve model capacity and inference performance. MoE models typically activate only a small number of experts in a single inference step, but all expert parameters still need to be reliably stored and supplied to the computing cores promptly after the routing is determined. This causes the system to simultaneously face multiple constraints, including memory capacity, expert parameter supply bandwidth, cross-node access latency, and resource utilization efficiency. Current industry deployments commonly employ paths such as high-bandwidth local memory residency, expert parallel clusters, or external memory expansion. However, each solution can only address some of the problems and cannot simultaneously meet the five requirements of large capacity, low latency, high bandwidth, high computing power utilization, and elastic scalability.

[0003] The current industrial deployment mainly adopts the following solutions: 1. On-chip UMA unified memory architecture: Unified Memory Architecture (UMA) integrates heterogeneous computing cores such as CPUs, GPUs, and NPUs onto a single SoC or package, enabling each computing unit to share unified physical memory and a consistent virtual address space. Its main advantage lies in eliminating explicit copying between host memory and device memory, achieving lower access latency through on-chip interconnects and high-bandwidth packaged memory. It is suitable for single-machine deployments of large edge models, workstation inference, graphics computing, and small to medium-sized models. Representative product forms include high-end SoCs using unified memory, AI PC processors, and heterogeneous computing platforms integrating CPUs, GPUs, and NPUs.

[0004] 2. Expert parallel cluster based on RDMA / IB: Expert Parallelism (EP) typically distributes the weights of different experts across different GPUs or server nodes. After the MoE router identifies the target expert based on token characteristics, it performs collective communication via InfiniBand, RoCE, or Ethernet RDMA, such as All-to-All and All-to-All-V, to send the token to the expert's node and retrieve the computation results. This approach has been widely used in multi-GPU training clusters and large-model online inference clusters, enabling horizontal scaling of total GPU memory capacity by increasing the number of servers.

[0005] 3. Traditional electrical signal CXL memory expansion solution: Traditional Compute Fast Link (CXL) memory expansion utilizes CXL Type-3 memory devices, CXL AIC expansion cards, or JBOM memory chassis. It provides uniformly addressable expansion capacity to the host via CXL.mem, and allows CXL switches and Fabric Manager to perform device enumeration, host binding, dynamic capacity allocation, and multi-host resource management. This solution is suitable for in-memory databases, virtualization, HPC, and large-memory AI services, expanding host memory without adding an equivalent number of computing chips.

[0006] The aforementioned prior art has the following drawbacks: At the physical layer, CXL has low latency but limited transmission distance, while RDMA / IB can support longer interconnect distances but usually introduces higher software and network processing overhead. When deployed across racks, it is difficult to simultaneously achieve memory access semantics, low access latency and transmission distance.

[0007] In terms of architecture, GPU servers bind high-value computing cores to local memory, requiring the addition of complete computing nodes to accommodate all expert parameters, even if the sole purpose is to expand capacity. MoE activates only a small number of experts at each step, resulting in a large number of unused parameters occupying expensive local memory for extended periods, leading to both idle memory and over-allocation of computing power.

[0008] In terms of scheduling, in traditional on-demand access schemes, remote expert weight reading is usually initiated only after the MoE routing result is determined, so that remote access enters the inference critical path; indiscriminate prefetching may cause bandwidth waste and cache pollution.

[0009] In terms of bandwidth, existing solutions perform a small amount of computation after moving the full weights to the computing nodes. The interconnect links become congested due to the ineffective transport, and the lack of near-memory computation means that filtering and compression cannot be completed on the data side.

[0010] In terms of hardware and software collaboration, in some general CXL memory extension software stacks, remote memory is mainly managed as general memory resources. The semantics of upper-layer MoE routing probability and expert heat are difficult to be directly pushed down to hardware or pooled controllers, thus limiting the ability to perform prefetching, priority scheduling and cache eviction based on inference state.

[0011] In summary, existing solutions cannot simultaneously achieve high capacity, low latency, high bandwidth, high computing power utilization, and elastic scalability within the same architecture. Summary of the Invention

[0012] The purpose of this invention is to provide a memory access method and an inference system based on CXL pooled memory and optical interconnect to solve the above-mentioned problems existing in the prior art.

[0013] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a memory access method applied to an inference system based on CXL pooled memory and optical interconnect. The inference system based on CXL pooled memory and optical interconnect includes multiple lightweight UMA computing nodes, a CXL optoelectronic converged switching network, and a globally shared memory pool, comprising: When the lightweight UMA compute node performs attention calculation for the current Transformer layer during inference, it predicts the probability of each expert being activated through an early prediction network deployed locally on the lightweight UMA compute node. The lightweight UMA compute node obtains the available bandwidth of the currently available CXL optical links, determines a dynamic prefetch threshold based on the available bandwidth, includes experts whose activation probability is greater than the dynamic prefetch threshold into a candidate prefetch set, and sends asynchronous prefetch operations for the weight parameters of each expert in the candidate prefetch set to the global shared memory pool, so as to transfer the weight parameters of each expert in the candidate prefetch set from the global shared memory pool to the local cache of the lightweight UMA compute node; After the attention calculation of the current Transformer layer is completed, if there are target expert weights that were not prefetched, the lightweight UMA computing node initiates a synchronous read of the target expert weights and performs subsequent expert calculations after the target expert weights are returned. After the inference of the current Transformer layer is completed, the lightweight UMA compute node updates the popularity value of each expert in the local cache. When the local cache capacity reaches the capacity threshold, it marks the weight parameters corresponding to one or more experts as invalid and reclaims the cache space in order of popularity value from low to high, until the local cache usage is lower than the capacity threshold.

[0014] In one possible design, the method further includes: Initialize an inference system based on CXL pooled memory and optical interconnect; The initialization of the inference system based on CXL pooled memory and optical interconnect includes: Optical engine calibration is performed on lightweight UMA computing nodes, CXL optoelectronic converged switching network, and global shared memory pool; Device discovery and enumeration are performed using CXL.io, and information on switch ports, Type-3 memory devices, and their capabilities is obtained. Global HDM address space allocation is performed on lightweight UMA computing nodes, CXL optoelectronic converged switching network, and global shared memory pool; Load the MoE large language model and establish a cache consistency catalog for the CXL optoelectronic converged switching network.

[0015] In one possible design, after initializing the inference system based on CXL pooled memory and optical interconnects, the method further includes: The global shared memory pool dynamically stores the expert's weight parameters in the hot data layer, warm data layer, and cold data layer based on the expert's access frequency, prediction hit rate, weight size, and service level. The pooling controller generates a capacity adjustment strategy based on the concurrent number of instances of each MoE large language model in the system, the growth rate of KV Cache, and the link utilization. Based on the capacity adjustment strategy, it performs dynamic capacity allocation or release on DCD and maintains the corresponding address mapping and access permissions. When the throughput of the inference system based on CXL pooled memory and optical interconnect is insufficient, a lightweight UMA computing node is added and a shared mapping is established between it and the model weights of the existing MoE large language model.

[0016] In one possible design, the method further includes: The CXL optoelectronic converged switching network allocates at least two physically independent optical paths to each key node; When the CXL optoelectronic converged switching network detects an increase in the bit error rate, a decrease in optical power, or a link interruption in any optical link, it stops sending new transactions to the faulty optical link and retransmits or recovers the unfinished request on the backup optical link. When no more than NK memory modules fail, the pooling controller or near-memory computing unit in the storage pool uses the remaining data and check shards to recover the missing content and rebuild it to the backup module. The weight parameters of the experts and the cache consistency catalog in the storage pool are distributed in multiple physically independent memory modules using RS(N,K) erasure coding or equivalent redundancy coding. N represents the total number of shards and K represents the number of data shards. When a memory region in any storage pool is migrated, the pooling controller sets a transition state between the old address mapping and the new address mapping. During the validity of the transition state, all concurrent write requests to the memory region in either storage pool are blocked, and the controller switches to the new mapping after the migration is complete.

[0017] In one possible design, the probability of the expert being activated is: ,in, This represents the probability that the i-th expert is predicted to be activated by the early prediction network, given the input token features X. This represents the event that the i-th expert is activated. Let X represent the weight matrix of the early prediction network, and let X represent the feature vector of the input token. This represents the bias vector of the early prediction network. Represents the temperature parameter, Softmax(·) i This represents the Softmax activation function.

[0018] In one possible design, the dynamic prefetch threshold determined based on available bandwidth is: ,in, Indicates the basic threshold. This represents the congestion penalty coefficient. This indicates the available bandwidth of the currently available CXL optical link. This indicates the maximum theoretical bandwidth of the currently available CXL optical link.

[0019] Secondly, the present invention provides an inference system based on CXL pooled memory and optical interconnect, including multiple lightweight UMA computing nodes, a CXL optoelectronic fusion switching network and a global shared memory pool, wherein the CXL optoelectronic fusion switching network is communicatively connected to the multiple lightweight UMA computing nodes and the global shared memory pool respectively. The lightweight UMA compute node is used to predict the probability of each expert being activated by an early prediction network deployed locally on the lightweight UMA compute node when performing attention calculations for the current Transformer layer during inference. Obtain the available bandwidth of the currently available CXL optical links, determine the dynamic prefetch threshold based on the available bandwidth, include experts whose activation probability is greater than the dynamic prefetch threshold into the candidate prefetch set, and send asynchronous prefetch operations for the weight parameters of each expert in the candidate prefetch set to the global shared memory pool, so as to transfer the weight parameters of each expert in the candidate prefetch set from the global shared memory pool to the local cache of the lightweight UMA computing node; After the attention calculation of the current Transformer layer is completed, if there are target expert weights that were not prefetched, a synchronous read of the target expert weights is initiated, and subsequent expert calculations are performed after the target expert weights are returned; and After the inference of the current Transformer layer is completed, the popularity value of each expert in the local cache is updated. When the local cache capacity reaches the capacity threshold, the weight parameters corresponding to one or more experts are marked as invalid and the cache space is reclaimed in order of popularity value from low to high, until the local cache usage is lower than the capacity threshold.

[0020] In one possible design, the CXL optoelectronic converged switching network adopts a two-layer topology or a star topology.

[0021] In one possible design, the CXL optoelectronic converged switching network adopts a Spine-Leaf topology. The CXL optoelectronic converged switching network includes a Leaf switching layer and a Spine switching layer. The Leaf switching layer is used to aggregate lightweight UMA computing nodes or global shared memory pool ports, and the Spine switching layer is used to complete cross-Leaf routing and multi-path forwarding.

[0022] The present invention has the following beneficial effects: 1. Architectural cost dimension: Reduce the total cost of ownership caused by bundling memory copies with computing resources.

[0023] This invention stores all expert weight parameters in a global shared memory pool. The lightweight UMA compute node only retains hot data and runtime cache. In scenarios where expert weight parameters account for a major proportion of the total memory and multiple inference replicas share the same model, duplicate weight replicas can be reduced. This centralized storage is accomplished by the global shared memory pool, which helps reduce multiple capacity fragments, cross-board management states, and duplicate pool control logic, and makes the capacity, power consumption, heat dissipation, and maintenance boundaries of the pooled memory more centralized.

[0024] 2. Transmission delay dimension: Reduce cross-node expert access from network message semantics to memory transaction semantics.

[0025] By using the CXL optoelectronic converged switching network, this invention maintains the CXL.mem memory access semantics in cross-rack deployment and parallelizes expert weight prefetching and Attention calculation based on the early prediction network. When the prediction is successful, the corresponding expert weight can be directly read from the local cache after the actual route is completed, thereby reducing the obstruction of the inference critical path by the acquisition of remote weights.

[0026] 3. Computing power utilization dimension: Reduce pipeline bubbles caused by expert waiting and improve MFU.

[0027] Under the condition that the prefetch time does not exceed the Attention calculation window and the candidate hit rate is high, the expert weight parameter supply delay can be partially or completely covered by the Attention calculation window, thereby reducing pipeline bubbles caused by expert waiting, which is conducive to improving the model's floating-point operation utilization and the effective throughput of concurrent inference.

[0028] Furthermore, by employing a scalable global shared memory pool, the available capacity of model weights can be expanded by adding memory modules or pooling resources, thereby overcoming the limitations of local memory capacity on a single computing node.

[0029] Furthermore, when adding new computing nodes, there is no need to copy the entire model; only a shared address mapping is established, reducing the risk of interruption for large-scale online inference services and improving overall availability. Attached Figure Description

[0030] Figure 1 A block diagram of an inference system based on CXL pooled memory and optical interconnect provided for an embodiment of this application; Figure 2 A block diagram illustrating a global shared memory pool provided in an embodiment of this application; Figure 3 A flowchart of a memory access method provided in an embodiment of this application. Detailed Implementation

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0032] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0033] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0034] It should be understood that specific details are provided in the following description to facilitate a complete understanding of the exemplary embodiments. However, those skilled in the art will understand that the exemplary embodiments can be implemented without these specific details. For example, the system may be shown in block diagrams to avoid obscuring the example with unnecessary details. In other instances, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the exemplary embodiments.

[0035] Please see Figure 1 The first aspect of this application provides an inference system based on CXL pooled memory and optical interconnect, including multiple lightweight UMA computing nodes, a CXL optoelectronic fusion switching network and a global shared memory pool, wherein the CXL optoelectronic fusion switching network is communicatively connected to the multiple lightweight UMA computing nodes and the global shared memory pool respectively.

[0036] The lightweight UMA compute node is used to predict the activation probability of each expert by using an early prediction network deployed locally on the lightweight UMA compute node when performing attention calculation of the current Transformer layer during inference; obtain the available bandwidth of the currently available CXL optical links, determine a dynamic prefetch threshold based on the available bandwidth, include experts whose activation probability is greater than the dynamic prefetch threshold into a candidate prefetch set, and send asynchronous prefetch operations for the weight parameters of each expert in the candidate prefetch set to the global shared memory pool to transfer the weight parameters of each expert in the candidate prefetch set from the global shared memory pool to the local cache of the lightweight UMA compute node; after the attention calculation of the current Transformer layer is completed, if there is a target expert weight that was not prefetched, a synchronous read is initiated for the target expert weight, and subsequent expert calculations are performed after the target expert weight is returned; and after the inference of the current Transformer layer is completed, the heat value of each expert in the local cache is updated, and when the local cache capacity reaches the capacity threshold, the cache space corresponding to one or more experts is reclaimed according to the heat value from low to high until the local cache occupancy is lower than the capacity threshold.

[0037] "Lightweight" refers to UMA nodes that do not reside with all expert weights for extended periods, but primarily retain computing units and local caches. "CXL Optoelectronic Convergence Switching Network" refers to setting up optical engines on the CXL node, switching port, and / or memory pool interface side, enabling CXL links to be transparently transmitted via optical fiber through electro-optical-electro-electric conversion.

[0038] The CXL optoelectronic converged switching network adopts a two-layer topology or a star topology. For example... Figure 1 As shown, the CXL optoelectronic converged switching network can adopt a Spine-Leaf topology, where lightweight UMA compute nodes or global shared memory pool ports are aggregated by the Leaf switching layer, and cross-Leaf routing and multi-path forwarding are completed by the Spine switching layer. The CXL optoelectronic converged switching network sets up optical engines on the CXL node, switching port, and / or global shared memory pool interface sides. These optical engines convert the high-speed electrical signals output from the CXL interface into optical signals, which are then transparently transmitted via fiber optic links and restored to electrical signals by the peer optical engine to maintain CXL transaction semantics. The optical engines are preferably implemented in a CPO (Content on Packaging) form, but can also be implemented in near-package, board-level integrated, or other optical interconnect forms capable of CXL electro-optical conversion. The global shared memory pool is preferably a dedicated physical pooling area uniformly managed by a pooling controller, such as a JBOM (Joint Memory Pooling Machine). In other embodiments, the global shared memory pool can also consist of physical memory resources distributed across multiple memory modules, chassis, or fault domains, presenting a unified resource namespace to the compute nodes.

[0039] like Figure 2 As shown, the global shared memory pool includes an optoelectronic conversion interface (which can be implemented by the CPO optical engine), a CXL pooling controller, a near-memory computing unit, a hot data layer (not shown), a warm data layer, and a cold data layer. The optoelectronic conversion interface is used to establish an optical link with the CXL optoelectronic converged switching network and complete the electro-optical-electro-electric conversion; the CXL pooling controller communicates with the near-memory computing unit and each data layer through a high-speed internal bus, and is responsible for the management of pooling resources, address mapping, and access permissions.

[0040] In one or more embodiments, the local cache of the lightweight UMA compute node may include on-chip SRAM / LLC and / or local HBM, GDDR, or UMA memory; the global shared memory pool may be further divided into a hot data layer, a warm data layer, and a cold data layer according to access frequency. As used herein, "local cache" refers to one or more storage tiers located on the compute node side, directly accessible by the compute node, and with access latency lower than that accessing the global shared memory pool via CXL optical interconnect.

[0041] Please see Figure 3 The second aspect of this application provides a memory access method, applied to... Figure 1The inference system based on CXL pooled memory and optical interconnect shown may include, but is not limited to, the following steps S301-S304.

[0042] Step S301. When the lightweight UMA computing node performs the attention calculation of the current Transformer layer during inference, it predicts the probability of each expert being activated by the early prediction network deployed locally on the lightweight UMA computing node.

[0043] In the MoE large model, the input token sequence (Where B is the batch size, S is the sequence length, and D is the hidden layer dimension) Typically, before reaching the MoE module of the current Transformer layer, it is first computed through the self-attention module.

[0044] Traditional architectures typically determine the target expert only after the Attention calculation is complete and the routing results are obtained. This invention employs an early prediction mechanism: at the start of the Attention calculation in the current Transformer layer, a lightweight UMA computation node acquires the Attention input features of the current layer and / or the output features of the previous layer, and feeds them into a lightweight Early Predictor (EP) network to estimate the activation probability of each expert in advance. The Early Predictor network can be implemented using linear mapping, a small multilayer perceptron (MLP), or other lightweight models.

[0045] In one or more embodiments, the early prediction network may employ offline supervised training, using the expert selection results generated by the real MoERouter for the training samples as supervision labels to learn the mapping between input features and expert activation probabilities. During the inference phase, the early prediction results are used to generate a candidate prefetch set, but do not replace the final routing results of the real MoE Router.

[0046] In one or more embodiments, the probability of an expert being activated is: ,in, This represents the probability that the i-th expert is predicted to be activated by the early prediction network, given the input token features X. This represents the event that the i-th expert is activated. Let X represent the weight matrix of the early prediction network, and let X represent the feature vector of the input token. This represents the bias vector of the early prediction network. Represents the temperature parameter, Softmax(·) i This represents the Softmax activation function.

[0047] Step S302. The lightweight UMA compute node obtains the available bandwidth of the currently available CXL optical links, determines the dynamic prefetch threshold based on the available bandwidth, includes experts whose activation probability is greater than the dynamic prefetch threshold into the candidate prefetch set, and sends asynchronous prefetch operations for the weight parameters of each expert in the candidate prefetch set to the global shared memory pool, so as to transfer the weight parameters of each expert in the candidate prefetch set from the global shared memory pool to the local cache of the lightweight UMA compute node.

[0048] Based on the probability of each expert being activated, the lightweight UMA compute node decides whether to initiate a prefetch request to the global shared memory pool. To control the prefetch scale according to the link congestion status, this invention calculates a dynamic prefetch threshold based on the available bandwidth of the current CXL optical link.

[0049] The dynamic prefetch threshold determined based on available bandwidth can be expressed as: ,in, Indicates the basic threshold. This represents the congestion penalty coefficient. This indicates the available bandwidth of the currently available CXL optical link. This represents the maximum theoretical bandwidth of the currently available CXL optical links. When the network is relatively idle, the dynamic prefetch threshold is low, and the system tends to expand the candidate prefetch set; when the network is congested, the dynamic prefetch threshold increases, and only expert weight parameters with higher activation probabilities are prefetched.

[0050] For the i-th expert, if the probability of the expert being activated is greater than the dynamic prefetch threshold, the lightweight UMA compute node preferably initiates an asynchronous or non-blocking read based on the address range corresponding to the weight parameters of the i-th expert through the CXL DMA (Direct Memory Access) engine, and transfers the weight parameters to the local cache of the lightweight UMA compute node via the CXL.mem access path. In other embodiments, the processor can also trigger a remote CXL.mem read through load / store or other memory access mechanisms.

[0051] In one or more embodiments, for low computational density operations such as Embedding table lookup, sparse aggregation, KV cache filtering, and weight rearrangement, even prefetching the original data may consume too much bandwidth. Therefore, these operations can be offloaded to the near memory side of the global shared memory pool for execution according to the computational load.

[0052] In one or more embodiments, compute nodes can issue tasks such as embedding lookup, sparse aggregation, KV cache filtering, or weight rearrangement to the PNM unit via custom or vendor-extended control operations. These control operations are used to carry task descriptions or opcodes in the CXL system and are not limited to standard CXL.mem commands. After receiving the task, the global shared memory pool uses the PNM unit to read local data on the memory side and perform lookup, filtering, compression, aggregation, or rearrangement. Only the processing results or a compressed subset of data is returned to the lightweight UMA compute node, thereby reducing the amount of raw data transmitted via the CXL link.

[0053] Step S303. After the attention calculation of the current Transformer layer is completed, if there are target expert weights that were not prefetched, the lightweight UMA computing node initiates a synchronous read of the target expert weights and performs subsequent expert calculations after the target expert weights are returned.

[0054] In this embodiment, at the start of the current Transformer layer, attention calculation for the current Transformer layer is initiated, and simultaneously, an early prediction network is started to predict the activation probability of each expert. Once the early prediction network completes its prediction (in a very short time), an asynchronous prefetch request for the weight parameters of the experts in the current Transformer layer is initiated. The lightweight UMA computing node continuously performs high-load attention calculations, while the CXL optoelectronic fusion switching network stores the acquired expert weight parameters in its local cache (HBM / GDDR / UMA) in the background.

[0055] After the Attention calculation is completed, an accurate routing vector is generated. At this point, it is checked whether the required expert weights have arrived in the local cache. When the prefetch latency does not exceed the Attention calculation window, the latency of acquiring remote weights can be partially or completely covered by the Attention calculation window.

[0056] After the attention calculation of the current Transformer layer is completed, if there are target expert weights (weight parameters of the missed experts) that were not prefetched, the system enters the on-demand reading path. The lightweight UMA computing node initiates synchronous reading of the target expert weights and continues to perform expert calculations after the target expert weights are returned.

[0057] Step S304. After the inference of the current Transformer layer is completed, the lightweight UMA compute node updates the popularity value of each expert in the local cache, and when the local cache capacity reaches the capacity threshold, the weight parameter corresponding to the expert with the lowest popularity value is marked as invalid and the corresponding cache space is reclaimed.

[0058] During inference, the local cache capacity of the lightweight UMA compute node is limited, necessitating cache management of used expert weight parameters. The local cache may include on-chip SRAM / LLC and / or local HBM, GDDR, or UMA memory. To avoid frequent swapping in and out, this invention maintains a local LRU (Least Recently Used) cache eviction queue based on popularity decay on the lightweight UMA compute node, and maintains popularity values ​​for expert weight parameters residing in the local cache. Each time a corresponding expert is selected for inference, its popularity value is increased, and then decayed after each inference step. When the local cache capacity reaches a threshold, the system preferentially evicts one or more expert weight parameters according to their popularity values ​​from low to high, until the local cache occupancy falls below the threshold, and then releases the corresponding cache entries or space. Since expert weight parameters are typically read-only during inference, the eviction process does not require writing weight data back to a remote global shared memory pool, reducing reverse transfer overhead.

[0059] In large-scale distributed interconnection systems, optical link interruptions and computing node or memory module failures may occur. Therefore, this application may also set up a fault tolerance mechanism, which may include, but is not limited to, the following steps S401-S404.

[0060] Step S401. The CXL optoelectronic converged switching network allocates at least two physically independent optical paths to each key node.

[0061] Step S402. When the CXL optoelectronic converged switching network detects an increase in the bit error rate, a decrease in optical power, or a link interruption in any optical link, it stops sending new transactions to the faulty optical link and retransmits or recovers the unfinished request on the backup optical link.

[0062] Step S403. When no more than NK memory modules fail, the pooling controller or near-memory computing unit in the storage pool uses the remaining data and check shards to recover the missing content and rebuild it to the spare module.

[0063] In this context, the weight parameters of experts and the cache consistency catalog in the storage pool are distributed in multiple physically independent memory modules using RS(N,K) erasure coding or equivalent redundancy coding, where N represents the total number of shards and K represents the number of data shards.

[0064] Step S404. When a memory region in any storage pool is migrated, the pooling controller sets a transition state between the old address mapping and the new address mapping. During the validity of the transition state, all concurrent write requests to the memory region in any storage pool are blocked, and the controller switches to the new mapping after the migration is completed.

[0065] In one or more embodiments, prior to memory access, it is also necessary to initialize the inference system based on CXL pooled memory and optical interconnect. Initializing the inference system based on CXL pooled memory and optical interconnect may include, but is not limited to, the following steps S501-S504.

[0066] Step S501. Perform optical engine calibration on the lightweight UMA computing node, CXL optoelectronic fusion switching network, and global shared memory pool.

[0067] After the lightweight UMA computing node, CXL optoelectronic fusion switching network, and global shared memory pool are powered on, a continuous wave laser source provides a stable light source to each CPO module. Firmware calibration of modulator bias, receiver sensitivity, CDR, and FEC parameters can be controlled, and channel alignment is completed through TS1 / TS2 training sequences. The link enters a usable state after reaching the target bit error rate.

[0068] Step S502. Perform device discovery and enumeration through CXL.io, and obtain information on the switch port, Type-3 memory device and its capabilities.

[0069] Each node's root port can discover switch ports and Type-3 memory devices via CXL.io. The pooling controller can collect device capacity, port speed, DCD (Dynamic Capacity Device) capability, PNM capability, and fault domain information, and can collaborate with system-level resource management entities such as Fabric Manager when needed. During enumeration, the globally shared memory pool is identified as a unified pooled memory resource entity. Multiple devices or modules within the pool are considered as capacity components within this entity, managed by the unified pooling controller, and presented with a unified resource namespace to the compute nodes.

[0070] Step S503. Perform global HDM address space allocation for the lightweight UMA computing node, CXL optoelectronic converged switching network, and global shared memory pool.

[0071] The pooling controller divides contiguous or interleaved physical regions from the global shared memory pool based on model instances and tenant requirements, and coordinates with the host-side resource management component to configure the device-side and / or host-side HDM Decoder to establish corresponding address mappings. The model weight region is set to read-only shared; the KV Cache and state region are set to session-level isolation by default; multi-node write-shared KV Cache is an optional implementation and is only enabled when the corresponding consistency mechanism is available.

[0072] Step S504. Load the MoE large language model and establish the cache consistency directory for the CXL optoelectronic converged switching network.

[0073] In one or more embodiments, after initializing the inference system based on CXL pooled memory and optical interconnect, the pooled memory resource management process can be entered. The pooled memory resource management process may include, but is not limited to, the following steps S601-S603.

[0074] Step S601. The global shared memory pool dynamically stores the expert's weight parameters in the hot data layer, warm data layer, and cold data layer according to the expert's access frequency, prediction hit rate, weight size, and service level.

[0075] Specifically, the weight parameters of high-frequency experts can be stored in the low-latency hot data layer, the weight parameters of regular experts can be stored in the warm data layer, and the weight parameters or checkpoints of long-tail experts can be placed in the cold data layer.

[0076] Step S602. The pooling controller generates a capacity adjustment strategy based on the concurrent instance count of each MoE large language model in the system, the KV Cache growth rate, and the link utilization. Based on the capacity adjustment strategy, it performs dynamic capacity allocation or release on DCD and maintains the corresponding address mapping and access permissions.

[0077] In one implementation, the pooling controller can work with Fabric Manager to manage related resources.

[0078] Step S603. When the throughput of the inference system based on CXL pooled memory and optical interconnect is insufficient, add a lightweight UMA computing node and establish a shared mapping between it and the model weights of the existing MoE large language model.

[0079] In one or more embodiments, the pooling controller can also copy reproducible read-only expert fragments to different fault domains or memory channels based on port queues, memory channel utilization, and hotspot expert distribution, and control the CXL optoelectronic converged switching network to perform multi-path routing for read requests.

[0080] The memory access method and inference system based on CXL pooled memory and optical interconnect provided by this invention have the following beneficial effects.

[0081] 1. Architectural cost dimension: Reduce the total cost of ownership caused by bundling memory copies with computing resources.

[0082] This invention stores all expert weight parameters in a global shared memory pool. The lightweight UMA compute node only retains hot data and runtime cache. In scenarios where expert weight parameters account for a major proportion of the total memory and multiple inference replicas share the same model, duplicate weight replicas can be reduced. This centralized storage is accomplished by the global shared memory pool, which helps reduce multiple capacity fragments, cross-board management states, and duplicate pool control logic, and makes the capacity, power consumption, heat dissipation, and maintenance boundaries of the pooled memory more centralized.

[0083] 2. Transmission delay dimension: Reduce cross-node expert access from network message semantics to memory transaction semantics.

[0084] By using the CXL optoelectronic converged switching network, this invention maintains the CXL.mem memory access semantics in cross-rack deployment and parallelizes expert weight prefetching and Attention calculation based on the early prediction network. When the prediction is successful, the corresponding expert weight can be directly read from the local cache after the actual route is completed, thereby reducing the obstruction of the inference critical path by the acquisition of remote weights.

[0085] 3. Computing power utilization dimension: Reduce pipeline bubbles caused by expert waiting and improve MFU.

[0086] Under the condition that the prefetch time does not exceed the Attention computation window and the candidate hit rate is high, the expert weight parameter supply delay can be partially or fully covered by the Attention computation window, thereby reducing pipeline bubbles caused by expert waiting, which is conducive to improving the utilization of computing resources and the effective throughput of concurrent inference.

[0087] 4. Memory capacity dimension: Breaking through the single-node packaging limit, forming a unified shared capacity across racks.

[0088] By employing a scalable global shared memory pool, the available capacity of model weights can be expanded by adding memory modules or pooling resources, thereby overcoming the limitations of local memory capacity on a single computing node.

[0089] 5. Elastic scaling and reliability: Supports independent expansion of cross-rack resources and on-demand recovery to reduce the risk of interruption.

[0090] When adding a new computing node, there is no need to copy the entire model; only a shared address mapping is established, which reduces the risk of interruption for large-scale online inference services and improves overall availability.

[0091] 6. Bandwidth overhead dimension: PNM transforms large blocks of raw data into smaller results for return.

[0092] For low computational density operations such as embedding lookup, sparse aggregation, KV cache filtering, and weight rearrangement, the PNM unit can directly read local data on the memory side and perform lookup, filtering, compression, aggregation, or rearrangement, returning only the processing results or a compressed subset of data. This reduces the amount of raw data transmitted via the CXL optical link and alleviates link congestion caused by concurrent access from hotspot experts.

[0093] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A memory access method applied to an inference system based on CXL pooled memory and optical interconnect, wherein the inference system based on CXL pooled memory and optical interconnect includes multiple lightweight UMA computing nodes, a CXL optoelectronic converged switching network, and a globally shared memory pool, characterized in that, include: When the lightweight UMA compute node performs attention calculation for the current Transformer layer during inference, it predicts the probability of each expert being activated through an early prediction network deployed locally on the lightweight UMA compute node. The lightweight UMA compute node obtains the available bandwidth of the currently available CXL optical links, determines a dynamic prefetch threshold based on the available bandwidth, includes experts whose activation probability is greater than the dynamic prefetch threshold into a candidate prefetch set, and sends asynchronous prefetch operations for the weight parameters of each expert in the candidate prefetch set to the global shared memory pool, so as to transfer the weight parameters of each expert in the candidate prefetch set from the global shared memory pool to the local cache of the lightweight UMA compute node; After the attention calculation of the current Transformer layer is completed, if there are target expert weights that were not prefetched, the lightweight UMA computing node initiates a synchronous read of the target expert weights and performs subsequent expert calculations after the target expert weights are returned. After the inference of the current Transformer layer is completed, the lightweight UMA compute node updates the popularity value of each expert in the local cache. When the local cache capacity reaches the capacity threshold, it marks the weight parameters corresponding to one or more experts as invalid and reclaims the cache space in order of popularity value from low to high, until the local cache usage is lower than the capacity threshold.

2. The memory access method according to claim 1, characterized in that, The method further includes: Initialize an inference system based on CXL pooled memory and optical interconnect; The initialization of the inference system based on CXL pooled memory and optical interconnect includes: Optical engine calibration is performed on lightweight UMA computing nodes, CXL optoelectronic converged switching network, and global shared memory pool; Device discovery and enumeration are performed using CXL.io, and information on switch ports, Type-3 memory devices, and their capabilities is obtained. Global HDM address space allocation is performed on lightweight UMA computing nodes, CXL optoelectronic converged switching network, and global shared memory pool; Load the MoE large language model and establish a cache consistency catalog for the CXL optoelectronic converged switching network.

3. The memory access method according to claim 2, characterized in that, After initializing the inference system based on CXL pooled memory and optical interconnect, the method further includes: The global shared memory pool dynamically stores the expert's weight parameters in the hot data layer, warm data layer, and cold data layer based on the expert's access frequency, prediction hit rate, weight size, and service level. The pooling controller generates a capacity adjustment strategy based on the concurrent number of instances of each MoE large language model in the system, the growth rate of KV Cache, and the link utilization. Based on the capacity adjustment strategy, it performs dynamic capacity allocation or release on DCD and maintains the corresponding address mapping and access permissions. When the throughput of the inference system based on CXL pooled memory and optical interconnect is insufficient, a lightweight UMA computing node is added and a shared mapping is established between it and the model weights of the existing MoE large language model.

4. The memory access method according to claim 1, characterized in that, The method further includes: The CXL optoelectronic converged switching network allocates at least two physically independent optical paths to each key node; When the CXL optoelectronic converged switching network detects an increase in the bit error rate, a decrease in optical power, or a link interruption in any optical link, it stops sending new transactions to the faulty optical link and retransmits or recovers the unfinished request on the backup optical link. When no more than NK memory modules fail, the pooling controller or near-memory computing unit in the storage pool uses the remaining data and check shards to recover the missing content and rebuild it to the backup module. The weight parameters of the experts and the cache consistency catalog in the storage pool are distributed in multiple physically independent memory modules using RS(N,K) erasure coding or equivalent redundancy coding. N represents the total number of shards and K represents the number of data shards. When a memory region in any storage pool is migrated, the pooling controller sets a transition state between the old address mapping and the new address mapping. During the validity of the transition state, all concurrent write requests to the memory region in either storage pool are blocked, and the controller switches to the new mapping after the migration is complete.

5. The memory access method according to claim 1, characterized in that, The early prediction network outputs the prediction score corresponding to each expert based on the input Token feature X, and normalizes the prediction score by the Softmax function to obtain the probability of each expert being activated. The probability of the i-th expert being activated is denoted as Pearly(Ei|X), where Ei represents the event of the i-th expert being activated.

6. The memory access method according to claim 1, characterized in that, The dynamic prefetch threshold determined based on available bandwidth is ,in, Indicates the basic threshold. This represents the congestion penalty coefficient. This indicates the available bandwidth of the currently available CXL optical links. This indicates the maximum theoretical bandwidth of the currently available CXL optical link.

7. An inference system based on CXL pooled memory and optical interconnect, characterized in that, It includes multiple lightweight UMA computing nodes, a CXL optoelectronic converged switching network, and a global shared memory pool; the lightweight UMA computing nodes include local caches, the CXL optoelectronic converged switching network includes an optical engine for CXL electrical signal to optical signal conversion, and the global shared memory pool includes a pooling controller and a pooled storage medium; the CXL optoelectronic converged switching network is communicatively connected to the multiple lightweight UMA computing nodes and the global shared memory pool respectively; The lightweight UMA compute node is used to predict the probability of each expert being activated by an early prediction network deployed locally on the lightweight UMA compute node when performing attention calculations for the current Transformer layer during inference. Obtain the available bandwidth of the currently available CXL optical links, determine the dynamic prefetch threshold based on the available bandwidth, include experts whose activation probability is greater than the dynamic prefetch threshold into the candidate prefetch set, and send asynchronous prefetch operations for the weight parameters of each expert in the candidate prefetch set to the global shared memory pool, so as to transfer the weight parameters of each expert in the candidate prefetch set from the global shared memory pool to the local cache of the lightweight UMA computing node; After the attention calculation of the current Transformer layer is completed, if there are target expert weights that were not prefetched, a synchronous read of the target expert weights is initiated, and subsequent expert calculations are performed after the target expert weights are returned; and After the inference of the current Transformer layer is completed, the popularity value of each expert in the local cache is updated. When the local cache capacity reaches the capacity threshold, the weight parameters corresponding to one or more experts are marked as invalid and the cache space is reclaimed in order of popularity value from low to high, until the local cache usage is lower than the capacity threshold.

8. The inference system based on CXL pooled memory and optical interconnect according to claim 7, characterized in that, The CXL optoelectronic converged switching network adopts a two-layer topology or a star topology.

9. The inference system based on CXL pooled memory and optical interconnect according to claim 8, characterized in that, The CXL optoelectronic converged switching network adopts a Spine-Leaf topology. The CXL optoelectronic converged switching network includes a Leaf switching layer and a Spine switching layer. The Leaf switching layer is used to aggregate lightweight UMA computing nodes or global shared memory pool ports, and the Spine switching layer is used to complete cross-Leaf routing and multi-path forwarding.