Expert weight storage method, electronic device, and computer-readable storage medium

Abstract

This disclosure provides an expert weight storage method, electronic device, and program product, relating to the field of artificial intelligence technology. The method includes: acquiring expert data units included in a hybrid expert large language model; determining the correlation strength metric between each expert data unit; dividing each expert data unit into multiple expert clusters based on the correlation strength metric; wherein each expert cluster includes at least one expert data unit; and storing the expert weights of the hybrid expert large language model in an auxiliary memory based on the multiple expert clusters, such that expert data units within the same expert cluster occupy consecutive or adjacent physical address ranges. This disclosure addresses the problem in the prior art where reading the expert weights of a MoE large language model stored in NAND flash memory requires numerous random read operations, resulting in high random read overhead.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to an expert weight storage method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In Mixture of Experts (MoE) large language models, a single processing step typically activates only a portion of the expert feedforward networks. Compared to dense models, MoE structures can scale parameter size with lower single-step computation, but the large number and size of expert weights often prevent them from being fully resident in high-speed memory, such as video random access memory (VRAM), on terminal devices or discrete graphics card platforms. To run large-scale models with limited computational and storage resources, it is usually necessary to store the large number of expert weights in auxiliary storage such as RAM (DRAM) or NAND flash memory, and dynamically load the required expert weights into the cache during inference based on routing decisions.

[0003] When a large number of expert weights are placed in DRAM or NAND media, expert access during inference exhibits characteristics of sparseness, non-uniformity, task-relatedness, and variation with inter-layer routing. In related technologies, expert weights in NAND are typically stored according to fixed rules such as expert number order, model file order, or write time order. However, the activation patterns of experts during the inference process of the MoE large language model are sparse and dynamic. This results in the accessed expert weights being dispersed across different pages, blocks, planes, dies, or channels in the physical medium. Reading requires initiating a large number of small-granularity random read operations, and significant read amplification occurs due to frequent crossing of page and block boundaries, leading to huge random read overhead. Summary of the Invention

[0004] In view of this, embodiments of the present disclosure provide an expert weight storage method, an electronic device, and a computer-readable storage medium to solve the problem in the prior art that when reading the expert weights of a MoE large language model stored in NAND, a large number of random read operations are required, resulting in high random read overhead.

[0005] Firstly, this disclosure provides a method for storing expert weights, including: Obtain the expert data units included in the hybrid expert large language model; Determine the correlation strength metric between each of the expert data units, and divide each expert data unit into multiple expert clusters based on the correlation strength metric; wherein each expert cluster includes at least one expert data unit. The expert weights of the hybrid expert large language model are stored in the auxiliary memory based on the multiple expert clusters, so that the expert data units of the same expert cluster occupy a continuous or adjacent physical address range.

[0006] Secondly, this disclosure provides an electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores at least one computer program that can be executed by the at least one processor, the at least one computer program being executed by the at least one processor to enable the at least one processor to perform the expert weight storage method as described in the first aspect.

[0007] Thirdly, this disclosure provides a computer program product, which includes a computer program that, when run in a processor, implements the expert weight storage method described in the first aspect.

[0008] The embodiments provided in this disclosure address the issue that when a hybrid expert large language model is deployed on the edge or in a resource-constrained environment, the expert weights stored in auxiliary storage such as NAND flash memory are often arranged in a fixed order, ignoring the access relationships between experts. This leads to experts that are frequently activated or accessed consecutively being scattered in the physical medium, causing a large number of random reads, page skipping, and read amplification problems. A method for storing expert weights is proposed. This method obtains the correlation strength metric between each expert data unit and divides the expert data unit into multiple expert clusters accordingly. This ensures that the expert weights of the same cluster occupy a continuous or adjacent physical address range in the auxiliary storage, thereby transforming scattered random reads into concentrated consecutive reads. This clustered storage layout based on access relationships effectively reduces random reads, page skipping, and read amplification, improves bandwidth utilization and data transfer efficiency from auxiliary storage to computing memory, and makes the loading process of expert weight data highly compatible with the actual inference access patterns of the hybrid expert large language model. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0010] Figure 1 The diagram shown is a flowchart of the expert weight storage method in an embodiment of this disclosure.

[0011] Figure 2The diagram shown is a schematic representation of the construction process of the expert dependency graph in an embodiment of this disclosure.

[0012] Figure 3 The diagram shown is a schematic diagram of the sequential placement of the NAND hierarchical structure in an embodiment of this disclosure.

[0013] Figure 4 The diagram shown illustrates the process of cluster-based batch reading and continuous prefetching during inference in this embodiment of the present disclosure.

[0014] Figure 5 The diagram shown is a schematic representation of the overall process of expert dependency graph and NAND sequential layout in an embodiment of this disclosure.

[0015] Figure 6 The diagram shown illustrates the process of maintaining the layout metadata table in an embodiment of this disclosure.

[0016] Figure 7 The diagram shown is a structural schematic of an electronic device in an embodiment of this disclosure. Detailed Implementation

[0017] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0018] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0019] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0020] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0021] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0022] Hybrid expert large language models contain a large number of expert weights. On edge devices or platforms with limited video memory, these expert weights often cannot reside entirely in high-speed storage and need to be stored in secondary storage and dynamically loaded during the inference phase. However, if the storage layout of expert weights in secondary storage does not consider the actual access relationships between experts, expert data units that are frequently activated or accessed consecutively may be physically scattered, leading to a large number of small-granularity random reads, page crossings, and read amplification. Read amplification refers to the additional NAND read operations or data movement caused by the scattered placement of expert weights, page boundary crossings, block fragmentation, or reading unnecessary data. How to reduce the random read overhead and read amplification during the loading of expert weights from secondary storage is the technical problem that this application embodiment needs to solve.

[0023] Analysis revealed that in related technologies, expert weights are typically stored in secondary storage according to expert ID, model file order, or write time order. This layout is disconnected from the actual expert access patterns during inference. In actual inference, the router of the MoE model selects and activates specific expert combinations based on the characteristics of the input lexical units. This means that certain experts are frequently activated together or accessed sequentially. However, since traditional storage layouts do not consider this access correlation, the physical locations of these frequently needed expert weights in secondary storage may be far apart. Each time the system loads, it has to initiate multiple small-granularity random read requests, traversing multiple physical pages and storage blocks, resulting in severe read amplification and reducing bandwidth utilization from secondary storage to high-speed storage.

[0024] To address the aforementioned issues, this application provides a method for storing expert weights, which can be widely deployed in various computing environments. For example, on edge devices (such as smartphones, tablets, and IoT terminals), workstations, and heterogeneous computing platforms equipped with discrete graphics cards, when the on-chip high-speed storage (such as VRAM) of the inference chip is insufficient to accommodate all the expert weights of a hybrid expert large language model, a large number of expert weights can be stored in auxiliary storage such as NAND flash memory and dynamically loaded into dynamic random access memory (DRAM) or VRAM during inference. At the storage device level, this method can be implemented in the controller firmware of an AI solid-state drive (AI SSD), where the SSD handles some layout management, address merging, and batch reading; it can also be implemented in a runtime system where the host and SSD work together, with the host responsible for association strength measurement and graph partitioning, and the SSD responsible for continuous reading and prefetching as instructed. In summary, this method is applicable to all scenarios requiring the storage of a large number of hybrid expert weights in NAND media and dynamic data transfer during inference, effectively reducing random reads and read amplification, and improving the data supply efficiency for edge-side large model inference.

[0025] like Figure 1 The diagram shown is a flowchart of the expert weight storage method provided in an embodiment of this application. The method mainly includes the following processing steps.

[0026] Step 101: Obtain the expert data units included in the MoE large language model.

[0027] The expert data unit mentioned here refers to the basic storage and scheduling object in the MoE large language model. For example, an expert data unit can correspond to an expert feedforward subnetwork, or it can correspond to a portion of an expert's weights, such as one or more matrix weight tensors, quantized weight blocks, compressed weight blocks, or expert-related auxiliary parameters. Organizing expert weights according to expert data units allows subsequent association strength measurement and cluster partitioning to be performed at an appropriate granularity. This captures expert-level access associations and allows for finer-grained layout optimization of different weight segments within an expert when needed.

[0028] Step 102: Determine the correlation strength metric between each expert data unit, and divide each expert data unit into multiple expert clusters based on the correlation strength metric. Each expert cluster includes at least one expert data unit.

[0029] Association strength metric is used to characterize the degree of association between two expert data units in their inference access behavior. The higher the association strength metric, the more likely the two expert data units are to be jointly activated or accessed sequentially during inference. By quantifying the degree of access association between expert data units, a basis can be provided for subsequently grouping highly associated expert data units into the same expert cluster, enabling the storage layout to adapt to the actual inference access pattern, rather than a fixed expert number or file order.

[0030] After obtaining the correlation strength metric between each expert data unit, these units can be divided into multiple expert clusters. Partitioning methods can include graph partitioning, community detection, spectral clustering, hierarchical clustering, heuristic merging, or hybrid grouping. The availability of multiple partitioning methods allows the system to flexibly select the most suitable partitioning strategy based on the size, structural characteristics, and deployment constraints of the expert dependency graph, thereby obtaining high-quality expert cluster partitioning results in different scenarios.

[0031] It should be noted that the step of determining the correlation strength metric between each expert data unit and dividing each expert data unit into multiple expert clusters based on the correlation strength metric can be performed offline or in the background.

[0032] Step 103: Based on multiple expert clusters, store expert weights in auxiliary memory so that expert data units of the same expert cluster occupy continuous or adjacent physical address ranges.

[0033] Therefore, highly relevant expert data units are clustered in physical storage, and during inference, the originally scattered multiple small-granular random reads can be transformed into batch reads of continuous or adjacent regions, reducing page spanning and block-range fragmentation, reducing read amplification, and improving bandwidth utilization from auxiliary storage to high-speed storage.

[0034] Expert data units within the same expert cluster occupy contiguous or adjacent physical address ranges. This means that expert weight data within the same expert cluster are arranged as contiguously as possible in the physical storage of auxiliary memory, with page alignment and local or regional contiguousness within blocks—a sequential layout. This layout does not require all expert data units to be in the same physical block, but it requires that related expert data units have a merging, prefetchable, or low-spanning storage range along the read path. By making highly correlated expert data units contiguous or adjacent in physical address, read requests for multiple expert data units within the same expert cluster during inference can be merged into one or a small number of consecutive read operations, reducing the number of random reads and page spans, and lowering read amplification overhead.

[0035] The auxiliary storage mentioned here refers to a non-high-speed resident storage device used to store the expert weights of the MoE large language model. For example, auxiliary storage can be a solid-state drive (SSD), such as an Artificial Intelligence Solid State Drive (AI SSD), or a non-volatile memory express (NVMe) storage device, or other storage based on non-volatile storage media. In other embodiments, auxiliary storage can also be a storage device with lower access speed but larger capacity than VRAM, such as Dynamic Random Access Memory (DRAM). The capacity of auxiliary storage is typically much larger than VRAM, capable of accommodating all or most of the expert weights of the MoE large language model, enabling the deployment of large-scale MoE large language models even on edge devices or discrete graphics card platforms with limited VRAM capacity.

[0036] In some embodiments, the auxiliary storage can be a NAND flash memory-based storage device. NAND flash memory has a hierarchical structure of pages, blocks, planes, dies, and channels. This hierarchical structure provides the physical basis for page alignment and contiguous layout within blocks for expert weights within the same expert cluster, while the multi-channel and multi-die-cell structure provides hardware support for parallel reading of different expert clusters. In some embodiments, the auxiliary storage can be an AI SSD based on NAND flash memory. AI SSDs integrate NAND flash memory media and controller logic, allowing some layout lookups, address range merging, and batch read operations to be performed on the SSD side, reducing computational and communication overhead on the host side. When laying out expert clusters in NAND flash memory, one or more of the following factors can be considered: page alignment, block range contiguity, plane locality, die-level distribution, channel parallelism, bad block bypass, wear leveling, and SSD internal address mapping strategies.

[0037] As explained above, the embodiments of this application drive the division of expert clusters through association strength metric values. The following further explains the method for determining association strength metric values.

[0038] In some embodiments, the association strength metric is determined based on association strength characterization information. This characterization information includes at least one of the following dimensions: number of co-activations, number of consecutive accesses, cross-layer transition probability, task relevance, and access time interval. Furthermore, the association strength characterization information also includes time decay coefficients for the aforementioned dimensions, which are used to determine the contribution of the association strength characterization information for the aforementioned dimensions to the association strength metric. By introducing multiple dimensions and time decay coefficients, the association strength metric can comprehensively reflect the access relationships between expert data units across multiple dimensions and can adapt to access patterns that evolve over time, avoiding excessive influence of outdated historical data on current layout decisions.

[0039] The co-activation count characterizes the frequency with which two expert data units are simultaneously activated by the router during inference. Expert data units with high co-activation counts are frequently needed simultaneously during inference. Assigning them to the same expert cluster and occupying a contiguous physical address range in auxiliary memory helps to load both expert data units simultaneously in a single read operation, reducing redundant independent reads.

[0040] The consecutive access count is used to characterize the frequency with which two expert data units are accessed sequentially during inference. Expert data units with consecutive access counts exceeding the access count threshold are read close to each other in time, are assigned to the same expert cluster, and occupy a continuous or adjacent physical address range in auxiliary memory. This allows the weights of two expert data units to be read at once, or the next expert data unit to be read sequentially after the previous one has been read, avoiding additional addressing overhead.

[0041] Cross-layer transition probability is used to characterize the probability of a data unit's access shifting from one layer to another within the MoE large language model. Cross-layer transition probability captures the continuity of expert access between layers in the MoE large language model. Expert data units with cross-layer transition probabilities higher than a probability threshold are considered highly correlated across layers. Assigning highly correlated expert data units to the same or adjacent expert clusters avoids additional random reads caused by frequent traversal of different physical regions during layer-by-layer inference.

[0042] Task relevance characterizes the degree to which two expert data units are accessed in a specific task, user session, or application domain scenario. Introducing the task relevance dimension allows the system to identify expert data unit groups that tend to be used together in specific scenarios, thereby enabling targeted optimization of expert cluster layout based on task characteristics when facing specific deployment scenarios.

[0043] The access time interval characterizes the temporal distance between two expert data units accessed sequentially during inference. Compared to focusing solely on the number of consecutive accesses in order of access, the access time interval further refines the degree of closeness in the temporal dimension. For example, if two expert data units are recorded as being activated sequentially in the access log, and the difference between their access timestamps falls within a preset short time window, then their access time interval is small, indicating a closer temporal access association between the two expert data units. Conversely, if the time interval is large, even if the two expert data units are adjacent in access order, their association may be relatively weak. Introducing the access time interval dimension allows the association strength metric to distinguish between seemingly continuous but actually loosely associated expert access sequences, capturing more granular temporal locality information and avoiding confusion between occasional adjacent accesses and high-frequency, tightly consecutive accesses. This provides a more accurate basis for the division of expert clusters, enabling the final generated expert clusters to more realistically reflect the dynamic memory access patterns during the inference process.

[0044] The time decay coefficient is used to apply a weighting that decreases over time to the statistics of each dimension in the association strength representation information, so that newer access behaviors contribute more to the association strength metric than earlier access behaviors. The expert activation patterns of the MoE large language model may evolve with changes in task distribution or user behavior. The time decay coefficient enables the association strength metric to adaptively reflect recent access patterns, avoiding outdated historical data from dominating layout decisions, thereby maintaining the matching degree between the storage layout and the current inference access pattern.

[0045] The association strength metric can be determined based on a weighted combination of association strength representations from multiple dimensions. The system normalizes the statistics for each dimension before weighted fusion to prevent any single statistic from excessively dominating the association strength metric result. Normalization and weighted fusion ensure that dimensional information with different dimensions and numerical ranges participates fairly in the calculation of the association strength metric, improving the balance and robustness of expert cluster partitioning. The weights of each dimension used in the association strength metric calculation are updated periodically, taking into account the time decay coefficient of each dimension. For example, the weight of a dimension is multiplied by its time decay coefficient over that period to obtain the updated weight of that dimension, and the sum of the weights of all dimensions is 1. For example, the association strength metric = w1 × normalized value of co-activation count + w2 × normalized value of consecutive access count + w3 × normalized value of cross-layer transition probability + w4 × normalized value of task relevance, where w1 is the weight value corresponding to the co-activation count, w2 is the weight value corresponding to the consecutive access count, w3 is the weight value corresponding to the cross-layer transition probability, and w4 is the weight value corresponding to the task relevance.

[0046] The above explains that the correlation strength metric can be determined based on the correlation strength characterization information. The following section further explains how to obtain the correlation strength characterization information.

[0047] In some embodiments, the process of determining the correlation strength metric between expert data units includes: collecting access logs of the MoE large language model over a specified time period; extracting correlation strength representation information between expert data units from the access logs; and determining the correlation strength metric between expert data units based on the correlation strength representation information. This systematic extraction process based on access logs makes the source of the correlation strength metric traceable and reproducible, and enables storage layout decisions to be driven by real-world reasoning behavior data.

[0048] The specified time period mentioned here can be a continuous or cumulative time window during the actual inference operation of the MoE large language model, or it can be a data collection period during the offline performance analysis phase or the online operation phase. This flexible time period selection allows the system to build an initial layout based on offline performance analysis data before deployment, or to continuously update the layout based on real-time inference data during the online operation phase, adapting to the needs of different deployment stages.

[0049] Access logs are used to record the expert access trajectories of the MoE large language model during inference. For example, each access log entry may include at least some of the following fields: request identifier, batch identifier, layer identifier, expert identifier, token position, access order, routing score (e.g., routing probability), model stage, task label, and timestamp. These multi-dimensional access log fields provide ample raw data support for subsequent extraction of association strength representation information across various dimensions, enabling the system to accurately characterize the access relationships between expert data units from multiple perspectives, such as co-activation, continuous access, cross-layer transfer, and task relevance.

[0050] Extracting association strength information from access logs can include, for example: counting the number of times two expert data units are simultaneously activated in the same request or batch to obtain the co-activation count; counting the number of times two expert data units are adjacent in access order to obtain the consecutive access count; calculating the expert access transfer relationship between different layers to obtain the cross-layer transfer probability; and calculating the degree of associated access between two expert data units based on task tags or user session categories to obtain the task relevance. This systematic extraction method ensures that all dimensions of association strength information can be automatically calculated from access logs, reducing manual intervention and improving the automation and accuracy of layout planning.

[0051] The above has explained how to determine the association strength metric. The following describes one implementation method for expert cluster partitioning based on the association strength metric.

[0052] In some embodiments, the process of dividing each expert data unit into multiple expert clusters includes: constructing an expert dependency graph, and then partitioning the expert dependency graph to obtain multiple expert clusters. By introducing the expert dependency graph as a graph structure, the association between each expert data unit is explicitly modeled as a weighted graph, enabling subsequent expert cluster partitioning to be performed using mature graph partitioning algorithms. This systematically aggregates highly correlated expert data units into the same cluster, providing a structured basis for sequential storage layout.

[0053] An expert dependency graph is a weighted graph with expert data units as nodes and measurable access relationships between them as edges. By abstracting the relationships between expert data units into a weighted graph structure, the problem of partitioning expert clusters can be transformed into a classic graph partitioning problem, thus allowing the reuse of mature algorithms and tools from the field of graph theory.

[0054] Graph nodes represent expert data units. For example, each graph node may correspond to an expert feedforward subnetwork or an expert weight segment. This correspondence between graph nodes and expert data units ensures consistency between the graph structure and the actual storage objects that need to be laid out in auxiliary storage.

[0055] Graph edges indicate that two connected graph nodes are related, meaning that the corresponding two expert data units have a measurable association in their reasoning and access behaviors. The existence of graph edges explicitly identifies which expert data units have access associations, providing structured information for subsequent graph partitioning algorithms to identify highly correlated expert sets. Specifically, for two expert data units that have common activation or continuous access, graph edges between these two data units are constructed in the expert dependency graph.

[0056] Graph edge weights represent the measure of the association strength between two graph nodes connected by an edge. A higher edge weight indicates a stronger access association between the corresponding two expert data units. By encoding the association strength measure into the graph structure, graph partitioning algorithms can prioritize retaining nodes connected by high-weight edges (association strength measures greater than a strength threshold) within the same subgraph (i.e., the same expert cluster), thereby achieving the aggregation of highly associated expert data units.

[0057] like Figure 2The diagram illustrates the construction process of the expert dependency graph in this embodiment. Access logs are first converted into graph edge features, then weighted and fused to form a weighted adjacency matrix and an expert dependency graph. The weighted adjacency matrix, a part of the expert dependency graph, is a mathematical representation of the graph edges and their weights. An adjacency matrix with a clear block structure indicates the existence of a highly correlated set of experts, which can serve as the basis for subsequent graph partitioning and sequential placement.

[0058] The above has explained how to construct an expert dependency graph. The following section will further explain the partitioning objectives of the graph.

[0059] In some embodiments, when partitioning the expert dependency graph, the partitioning can be performed according to a partitioning objective, resulting in multiple expert clusters. The partitioning objective may include at least one of the following: maximizing the graph edge weights within an expert cluster; minimizing the graph edge weights between different expert clusters; controlling the data size of a single expert cluster to not exceed a data threshold; controlling the number of expert data units within a single expert cluster to not exceed a quantity threshold; and grouping expert data units located at the same layer or at the association layer into the same expert cluster. By setting the above multi-dimensional partitioning objectives, the generated expert clusters achieve a balance between access correlation, storage resource consumption, and model structure adaptability, thereby providing better cluster partitioning results for subsequent sequential storage layout.

[0060] Maximizing the edge weights within an expert cluster implies having the highest possible access correlation between expert data units within the same cluster. By placing them adjacent in physical storage, inference can be performed by sequential reads when these expert data units are jointly activated or accessed consecutively, effectively reducing random reads.

[0061] Minimizing the graph edge weights between different expert clusters means minimizing the access correlation between expert data units in different expert clusters, thereby reducing the probability of needing to read related expert data units across different physical regions during inference and reducing the frequency of random cross-region readings.

[0062] Controlling the data size of a single expert cluster to not exceed the data threshold ensures that the expert weight of each expert cluster can be adapted to the physical storage structure of the auxiliary storage (such as block size, page group range, etc.), preventing the inability to place a single expert cluster completely in a contiguous physical area due to its excessive size. It also helps to control the amount of data read in a single batch, avoiding excessive consumption of read bandwidth.

[0063] Controlling the number of expert data units within a single expert cluster to not exceed a certain threshold helps maintain high correlation within the cluster while avoiding increased layout and scheduling complexity caused by excessively large cluster sizes. It also makes the granularity of cluster-by-cluster reading more controllable during runtime.

[0064] The reason for grouping expert data units located in the same layer into the same expert cluster is that expert data units in the same layer of the MoE large language model are usually selected by the router in the same inference step. Grouping them into the same expert cluster can ensure that experts in the same layer are physically continuous or adjacent, reducing cross-region reads in a single inference step.

[0065] Grouping expert data units located in the association layer into the same expert cluster is based on the fact that the association layer refers to the layer with a high probability of cross-layer transition during inference (i.e., the probability of cross-layer transition is higher than the probability threshold). When an expert in the previous layer is activated during inference and then a specific expert in the next layer needs to be accessed, grouping these cross-layer associated expert data units into the same cluster and storing them adjacently can reduce the additional random reads caused by crossing different physical regions during layer-by-layer inference.

[0066] Based on the above classification objectives, the ultimate goal can be achieved, namely, reducing the proportion of random reads during the runtime of the MoE large language model.

[0067] When partitioning the expert dependency graph according to the above-mentioned partitioning objectives, methods such as multi-level graph partitioning, community detection, spectral clustering, hierarchical clustering, learning-based grouping, or incremental merging based on heuristic rules can be employed. Multiple graph partitioning methods are available, allowing the system to select the most suitable partitioning algorithm based on the size and structural characteristics of the expert dependency graph, achieving a balance between partitioning quality and computational cost. The partitioning results of the expert dependency graph are mapped to the hierarchical structure of auxiliary memory, ensuring that weights within the same expert cluster are located in contiguous or adjacent regions as much as possible.

[0068] The method of dividing expert clusters has been explained above. The process of storing expert weights in auxiliary memory based on expert clusters will be explained below.

[0069] In some embodiments, when storing the expert weights of the MoE large language model in auxiliary memory based on multiple expert clusters, the descriptive information of each expert cluster is first obtained, and then the expert weights are stored in auxiliary memory according to the descriptive information of each expert cluster. By introducing the descriptive information of the expert clusters, the attributes such as the identifier, membership composition, and data size of the expert clusters are structurally represented, providing the necessary input information for the sequential layout planning in auxiliary memory, enabling the layout planner to determine the storage location and space allocation of each expert cluster based on this descriptive information.

[0070] The description information of an expert cluster includes the cluster identifier, the identifier of each expert data unit within the cluster, and the cluster size. The cluster identifier uniquely identifies an expert cluster, establishing a mapping between logical clusters and physical memory layout. This allows the runtime to quickly locate the physical address range of the corresponding expert cluster in auxiliary storage using the cluster identifier. The identifier of each expert data unit within the cluster indicates which expert data units are included in that cluster. This allows the runtime to quickly determine the expert cluster to which an activated expert data unit belongs upon receiving its identifier, and then perform batch readings by cluster. The cluster size represents the total size of the expert weight data for all expert data units within the cluster. This provides space requirement information for the layout planner to allocate contiguous or adjacent physical address ranges, and also provides a basis for the runtime to estimate the amount of data to be read and allocate temporary buffers when reading by cluster.

[0071] Furthermore, in some embodiments, the description information of expert clusters may also include the access popularity of the expert cluster and / or the association strength metric of the expert cluster. By introducing access popularity and association strength metrics, the system can implement differentiated storage strategies for expert clusters with different characteristics, enabling high-popularity or highly associated expert clusters to obtain better physical storage locations to ensure read performance, while low-popularity expert clusters can release high-quality storage resources through storage overhead optimization, thereby improving the overall storage efficiency and read performance of the auxiliary storage.

[0072] The access popularity of an expert cluster characterizes how frequently that cluster is accessed during inference. For example, access popularity can be determined based on the cumulative number of accesses to expert data units within the cluster, the access frequency per unit time, or recent access trends. Access popularity can have multiple status levels, such as hot, warm, cold, and fluctuating. Access popularity provides quantitative information on the frequency of expert cluster usage, enabling the system to identify the most frequently accessed expert clusters during inference and prioritize their storage layout optimization.

[0073] The association strength metric of an expert cluster is used to characterize the access association strength between the expert cluster as a whole and other expert clusters or expert data units within the cluster. The association strength metric of an expert cluster reflects the tightness of the association access of expert data units within the cluster. Expert data units within an expert cluster with high association strength are loaded together more frequently, have higher requirements for read continuity, and therefore need to be given priority in obtaining contiguous physical storage areas.

[0074] If the access frequency of an expert cluster exceeds a first access frequency threshold, the system prioritizes storing the expert weights of that cluster in a contiguous page area or a contiguous area within a block. High-access expert clusters are frequently accessed during inference; storing their expert weights in a contiguous page area or a contiguous area within a block ensures that they are loaded sequentially each time they are accessed, minimizing random reads and page skipping for these frequently accessed clusters, thus significantly improving overall inference performance.

[0075] If the association strength metric of an expert cluster is higher than the association strength threshold, the system will also prioritize storing the expert weights of that expert cluster in a contiguous page area or a contiguous area within a block. Expert data units within a high-association-strength expert cluster have high access correlation; storing them in a contiguous area allows for the effective merging of read requests for multiple expert data units in that cluster into a single contiguous read operation during inference, fully leveraging the advantages of a contiguous storage layout.

[0076] When the access frequency of an expert cluster is below the second access frequency threshold, the system optimizes the storage overhead of the expert weights for that cluster and stores them in a physical region with a priority lower than the priority threshold. Here, the second access frequency threshold is less than the first access frequency threshold. Low-access expert clusters are accessed less frequently during inference. Optimizing their storage overhead and storing them in a lower-priority physical region frees up high-quality contiguous storage areas for high-access or highly correlated expert clusters, while also reducing overall storage space usage. This improves the space utilization efficiency of auxiliary storage without significantly impacting inference performance. Storage overhead optimization can include, for example, compressing, quantizing, or packing expert weights to reduce the storage footprint of infrequently accessed expert weights, allowing auxiliary storage to accommodate more expert weights or freeing up space for other uses. The physical region with a priority lower than the priority threshold mentioned here refers to a storage region in auxiliary storage with lower read performance or access priority compared to contiguous pages or contiguous areas within a block.

[0077] Furthermore, in some embodiments, the description information of the expert cluster may also include an identifier of a target expert cluster that can be read in parallel with the current expert cluster. A target expert cluster refers to other expert clusters that are expected to be read simultaneously or adjacent to the current expert cluster during inference. Identifying expert cluster pairs that can be read in parallel provides a basis for distributing them to different physical read paths, avoiding resource contention caused by parallel read requests pointing to the same channel or the same wafer cell.

[0078] When there is a target expert cluster that can be read in parallel, the system stores the expert weights of that expert cluster and the corresponding target expert cluster in different channels or different die cells. Here, "channel" refers to an independent data transfer channel in the auxiliary memory, where each channel can perform read operations in parallel. Distributing the expert clusters that can be read in parallel across different channels allows multiple expert clusters to be read simultaneously using channel-level parallelism, shortening the overall loading time of the expert weights. "Different die cells" refers to different physical chip units in the auxiliary memory. Within the same channel, different die cells can also support a certain degree of parallel or interleaved reading. Distributing the expert clusters that can be read in parallel across different die cells allows for improved throughput even with limited channel resources, utilizing die-level parallel or interleaved reading capabilities.

[0079] like Figure 3 The diagram shown is a schematic diagram of the sequential placement of the NAND hierarchical structure in an embodiment of this disclosure. While keeping the expert weights within the same expert cluster continuous within the block, the system distributes different expert clusters that are expected to be read in parallel to different channels or different chip units, thereby reducing random reads while retaining the multi-channel parallel read capability inside the auxiliary memory.

[0080] The process of storing expert weights in auxiliary storage based on expert clusters has been explained above. The construction of the layout metadata table will be further explained below.

[0081] In some embodiments, after storing the expert weights of the MoE large language model in auxiliary storage based on multiple expert clusters, the system can also construct a layout metadata table. The layout metadata table is a structured data table used to associate and store the logical description information of expert clusters with their physical layout information in auxiliary storage. The layout metadata table centrally manages the originally scattered logical and physical information, enabling a complete mapping from expert identifiers to physical address ranges to be obtained through a single table query at runtime, avoiding the overhead of multiple scattered queries.

[0082] The layout metadata table includes descriptions of expert clusters and their physical layout information in auxiliary memory. The physical layout information includes the physical address range occupied by the expert data units of the expert cluster. For example, the physical address range may include information such as channel identifier, wafer cell identifier, plane identifier, block range, and page offset. This precise physical address range information, down to the channel, wafer cell, plane, block, and page levels, allows the runtime to accurately locate the physical position of expert weights in auxiliary memory and determine whether multiple read requests can be merged or whether page crossings are involved, thereby making optimal read scheduling decisions.

[0083] The layout metadata table can be stored internally in secondary storage (such as AI SSDs), in host memory, in the controller metadata area of ​​the secondary storage, or in a mirrored structure of both. This flexible choice of metadata storage location allows the system to optimize metadata access latency based on the specific deployment architecture. In AI SSD implementations, storing metadata on the SSD side allows at least some layout queries and address ranges to be performed on the SSD side, reducing unnecessary data round trips between the host and storage.

[0084] It should be noted that expert cluster partitioning and layout metadata table generation can both be completed offline or in the background. Furthermore, graph construction and partitioning can be performed periodically, or triggered under low load or after detecting significant changes in access patterns, to update expert clusters and layout metadata tables based on actual access changes. Some expert clusters can be adjusted incrementally based on changes in expert access logs, without needing to repartition expert clusters each time.

[0085] Furthermore, in some embodiments, the description information of the expert cluster may also include at least one of the following: updated version, storage overhead optimization information, recommended prefetch window, prefetch priority, and valid status identifier. By introducing these fields, the layout metadata table can support more granular runtime scheduling strategies and background layout maintenance operations.

[0086] The updated version represents the version of the expert cluster partitioning and storage layout. Versioning management allows the system to generate candidate layouts in the background and safely activate the new layout through version switching after data migration is complete, avoiding inconsistent layout states read during inference. New layout mappings can be generated in the background and activated through versioned metadata switching after data migration is complete.

[0087] Storage overhead optimization information describes the storage optimization status of the expert weights in an expert cluster. For example, storage overhead optimization information may include optimization statuses such as uncompressed, quantized, compressed, or packaged. This information enables the runtime to determine, after reading the expert weights, whether decompression, dequantization, or format conversion operations are needed, and allocates corresponding processing resources and DRAM temporary storage space accordingly.

[0088] The recommended prefetch window represents the recommended range of physical addresses for prefetching when reading the expert weights of the expert cluster. The recommended prefetch window provides directly usable address range guidance for continuous prefetching at runtime, enabling prefetch operations to quickly determine the physical range for prefetching without querying other metadata, thus reducing prefetch scheduling latency.

[0089] Prefetch priority indicates the priority of an expert cluster in runtime prefetch scheduling. Prefetch priority enables the system to prioritize the expert clusters that have the greatest impact on inference performance, thus optimizing the overall prefetch benefits, given limited read bandwidth and DRAM buffer resources.

[0090] A valid status flag indicates whether the layout status of the expert cluster is valid. For example, a valid status flag may include states such as valid, migrating, expired, or rollback only. The valid status flag allows the runtime to determine whether the current layout is in a migration transition period or has expired when querying layout metadata, avoiding reading data that is being migrated or using expired layout information, thus ensuring data consistency during the inference process.

[0091] The construction method and content of the layout metadata table have been explained above. The following section explains how to use the layout metadata table to perform batch pre-fetching of expert weights at runtime.

[0092] In some embodiments, the system first obtains the identifiers of prefetched expert data units that are currently active or in a candidate state. Currently active prefetched expert data units refer to those selected and activated by the router of the MoE large language model in the current inference step. Triggering prefetching based on currently active expert data units ensures that the expert weights required for the current inference step are loaded in a timely manner. Currently in a candidate state prefetched expert data units refer to those that have not yet been formally activated but are predicted to be needed by subsequent lexical units or subsequent layers based on router output or inference progress. Triggering prefetching based on candidate expert data units allows the system to preload potentially needed expert weights while performing computation in the current inference step, achieving time overlap between reading and computation and reducing the waiting time for expert weights.

[0093] The system then queries the layout metadata table to obtain the prefetch expert cluster where the prefetch expert data unit is located, and obtains the physical layout information of the prefetch expert cluster in the auxiliary storage. By querying the layout metadata table, the system can convert the expert identifier at the logical level into the cluster identifier and address range at the physical level, establishing a fast mapping path from inference logic to storage operations.

[0094] Finally, the system performs batch prefetching of expert weights based on the physical layout information of the prefetched expert clusters in auxiliary memory. Batch prefetching refers to reading the expert weights within a prefetched expert cluster from auxiliary memory to high-speed storage (e.g., DRAM or VRAM) in batches, based on the physical layout information of the prefetched expert clusters in auxiliary memory. For example, if multiple prefetched expert data units belong to the same prefetched expert cluster or adjacent expert clusters, the system can merge their physical address ranges and initiate one or more consecutive read operations. Batch prefetching at the cluster level combines multiple small-granularity random reads that would otherwise need to be initiated separately for each expert data unit into consecutive reads, reducing the number of read operations and page spans, reducing read amplification, and improving bandwidth utilization from auxiliary memory to high-speed storage.

[0095] In some embodiments, if the actually activated expert data unit is not within the prefetched expert cluster, the system can perform a bounded backoff read and record the event for subsequent updates to the graph edge weights and storage layout strategy of the expert dependency graph. Bounded backoff reads provide a fallback mechanism for prefetch misses, ensuring that inference is not interrupted by prefetch failure. Simultaneously, the recording of backoff events provides feedback data for subsequent incremental updates of graph weights and layout, enabling the storage layout to gradually adapt to the actual inference access pattern. The bounded backoff read mentioned here means that when a prefetch miss occurs, the system does not degenerate into an unrestricted global random read, but rather performs a fallback read within a limited and controllable candidate range. For example, the backoff range can be constrained to the local neighborhood of the currently missed expert in the expert dependency graph. For instance, the system can backoff read several neighboring nodes with the highest association strength metric with the expert in the graph structure, or read other expert data units belonging to the same upper-level parent cluster, thereby avoiding excessive computational overhead introduced by global traversal of the dependency graph. Alternatively, since the layout of the expert cluster itself has concentrated the weights of closely related experts in a continuous or adjacent range of physical addresses, even if the prefetch window fails to accurately cover the target expert, the physical location of the target expert in the auxiliary memory is likely to be adjacent to the prefetched area. The system only needs to perform supplementary reading within the limited adjacent range of the read physical address to complete the rollback without incurring the high random addressing overhead of cross-page, cross-block or even cross-channel.

[0096] like Figure 4The diagram illustrates the cluster-based batch reading and continuous prefetching process during inference in this embodiment. While the GPU performs expert computation on the current term, the NAND read queue can initiate continuous readings in parallel, such as reading the continuous page range corresponding to the current expert cluster. Simultaneously, the PCIe or memory transport link can temporarily store the read data in DRAM and then perform decompression, dequantization, or format conversion before transporting it to VRAM, thus preparing weight data in advance for the expert computation of the next term or the next layer. Therefore, the current computation window, the NAND read window, and the DRAM / VRAM transport window can form a pipelined overlap, reducing the idle time of the computation unit waiting for expert weights to load. Furthermore, in some embodiments, if an expert miss occurs, i.e., the actually activated expert is not covered by the current prefetch window, the system can trigger a bounded rollback mechanism to perform supplementary readings and record the rollback event. In this way, the system can overlap the continuous reading of auxiliary memory, DRAM temporary storage, data transfer and computing window of computing unit (such as graphics processing unit, GPU) while ensuring the continuity of inference, thereby reducing random read overhead and expert waiting latency, improving the data supply efficiency from NAND to DRAM or VRAM, and improving inference throughput and response speed.

[0097] like Figure 5The diagram illustrates the overall process of expert dependency graph and NAND sequential layout in this embodiment. The system first obtains the inference input of the hybrid expert large language model, including the expert data unit currently selected by the router, batch and token trajectories, and expert access logs. Based on these logs, an expert dependency graph is constructed. Specifically, the system can represent expert data units as graph nodes, establishing connections between two expert data units with common activation relationships or continuous access relationships as graph edges. The edge weights are determined based on the number of common activations, the number of continuous accesses, the cross-layer transition probability, and task relevance, thus forming a weighted graph structure reflecting expert access correlation. On this basis, the system performs graph partitioning on the expert dependency graph to obtain expert clusters and outputs the member composition and cluster identifier of each expert cluster. Subsequently, the system combines the physical hierarchy of NAND flash memory, such as pages, blocks, planes, dies, and channels, to perform sequential layout planning for each expert cluster. Highly correlated or frequently accessed expert clusters are preferentially mapped to continuous page areas, continuous areas within blocks, or different channels or different chip units suitable for parallel access, generating corresponding physical layout information and constructing a layout metadata table. The layout metadata table records the mapping relationship between expert data units and expert clusters, the mapping relationship between expert clusters and NAND physical address ranges, as well as information such as layout version, active status, and compression status. During runtime, the system can query the layout metadata table based on the active or candidate expert data units in the current inference phase, merge adjacent or consecutive physical address ranges, initiate batch prefetching or continuous reads by cluster, move expert weights from NAND to the DRAM temporary storage area, and, if necessary, perform decompression, dequantization, and other processing before further transferring them to VRAM for the computing unit to perform expert feedforward calculations. A sequential storage area for expert weights is formed within the AI ​​SSD or NAND medium, where different expert clusters can be distributed across different channels or regions according to their popularity, relevance, and parallel read requirements. This reduces random reads and read amplification while improving data transfer efficiency and inference throughput from NAND to DRAM or VRAM.

[0098] In some embodiments, telemetry data during the runtime of the MoE large language model is continuously collected and parsed. Based on the telemetry data, expert metadata, graph metadata of the expert dependency graph, cluster metadata of expert clusters, and physical layout metadata are maintained. Expert metadata includes expert identifier, layer identifier, weight size, compression status, quantization status, logical offset, current cluster identifier, and popularity status. Graph metadata includes edge type, edge weight, update time, statistical window, and normalization parameters. Cluster metadata includes cluster identifier, member expert list, cluster size, cluster popularity, NAND address range, prefetch priority, layout version, and validity status. Physical layout metadata includes channels, dies, planes, block range, page offset, continuity, bad block bypass status, and candidate layout versions.

[0099] like Figure 6 The diagram illustrates the process of maintaining the layout metadata table in this embodiment. The system can maintain a multi-level metadata structure covering experts, graph edges, expert clusters, and NAND physical layout to support a complete closed loop from runtime monitoring to background layout updates. During runtime, the system continuously collects and monitors telemetry information such as cluster hit rate, random read ratio, read amplification estimation, and rollback count. This telemetry information, combined with information such as time decay, new access edges, and task domain correction, is used to update graph weights, ensuring that the correlation strength metric between expert data units accurately reflects the actual access pattern at the current stage. Based on the updated edge weights, expert cluster repartitioning is performed. For example, based on cross-cluster access costs, cluster size constraints, and hot migration, it determines whether the existing expert cluster partitioning and storage layout still meet performance requirements, such as whether there is cross-cluster edge weight increase, cluster size exceeding limits, or hot migration. If adjustments are deemed necessary, the system can generate a new layout mapping in the background, such as replanning the continuous page layout, channel distribution scheme, and compression status adjustments, forming candidate layout versions. The new layout mapping can be safely activated via versioned metadata switching after data migration is complete, while retaining the old version mapping to support rollback and avoid inconsistent layout states read during inference. Data migration can be triggered under low load, and intra-block rearrangement and bad block bypass are performed during migration. After migration, a new layout metadata table is formed. Through the above methods, the system can form a closed-loop optimization mechanism from runtime telemetry, graph weight update, repartitioning determination, candidate layout generation, versioned switching, and background migration implementation. This allows the storage layout of expert weights to continuously evolve with changes in actual inference access patterns, maintaining low random read overhead and high data read efficiency.

[0100] In one exemplary embodiment, the MoE large language model is deployed on an edge device with a dedicated GPU, limited VRAM, system DRAM, and AI SSD. Expert weights not resident in VRAM are stored in the NAND medium of the AI ​​SSD. The system constructs an expert dependency graph based on historical access logs, divides highly correlated expert data units into clusters, and places weights within the same cluster into consecutive pages or adjacent block ranges. During inference, after the router outputs the currently active expert, the system reads the range by cluster and loads the corresponding expert weights into DRAM or VRAM.

[0101] In another exemplary embodiment, an initial expert dependency graph and NAND layout are first generated based on offline performance analysis data. During actual inference, new expert activation patterns are continuously recorded. When the runtime graph structure deviates significantly from the current NAND layout, candidate NAND layouts are generated in the background, and data migration and metadata version switching are performed under low load.

[0102] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic, and the execution order between steps is not limited to implementation according to step number.

[0103] like Figure 7 As shown, this disclosure also provides an electronic device, which includes: at least one processor 401; at least one memory 402; and one or more I / O interfaces 403 connected between the processor 401 and the memory 402; wherein the memory 402 stores one or more computer programs that can be executed by at least one processor 401, and the one or more computer programs are executed by at least one processor 401 to enable at least one processor 401 to execute the above-described expert weight storage method.

[0104] The modules in the aforementioned electronic devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0105] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described expert weight storage method.

[0106] As is known to those skilled in the art, the term computer storage medium includes media implemented in any method or technique for storing information, such as computer-readable program instructions, data structures, program modules, or other data.

[0107] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0108] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0109] Each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0110] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications or equivalent substitutions made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for storing expert weights, characterized in that, include: Obtain the expert data units included in the hybrid expert large language model; Determine the correlation strength metric between each of the expert data units, and divide each expert data unit into multiple expert clusters based on the correlation strength metric; wherein each expert cluster includes at least one expert data unit. The expert weights of the hybrid expert large language model are stored in the auxiliary memory based on the multiple expert clusters, so that the expert data units of the same expert cluster occupy a continuous or adjacent physical address range.

2. The expert weight storage method according to claim 1, characterized in that, The association strength metric is determined based on association strength characterization information; wherein the association strength characterization information includes at least one of the following dimensions: number of co-activations, number of consecutive accesses, cross-layer transition probability, task relevance, and access time interval; and the association strength characterization information also includes a time decay coefficient for the dimension, the time decay coefficient being used to determine the contribution of the association strength characterization information of the dimension to the association strength metric.

3. The expert weight storage method according to claim 2, characterized in that, Determining the correlation strength metric between each of the expert data units includes: Collect the access logs of the hybrid expert big language model over a specified time period; Extract the correlation strength representation information between each of the expert data units from the access log; Based on the correlation strength characterization information between each of the expert data units, a correlation strength metric value between each of the expert data units is determined.

4. The expert weight storage method according to claim 1, characterized in that, The step of dividing each expert data unit into multiple expert clusters based on the correlation strength metric between the expert data units includes: Construct an expert dependency graph; wherein the expert dependency graph includes graph nodes, graph edges connecting two graph nodes and graph edge weights, the graph nodes represent expert data units, the graph edges indicate that two connected graph nodes are related, and the graph edge weights represent the correlation strength measure between the two graph nodes connected by the graph edge; The expert dependency graph is partitioned to obtain multiple expert clusters.

5. The expert weight storage method according to claim 4, characterized in that, The expert dependency graph is partitioned to obtain multiple expert clusters, including: The expert dependency graph is partitioned according to the partitioning objective to obtain multiple expert clusters; The partitioning objective includes at least one of the following: Maximize the graph edge weights within the expert cluster; Minimize the graph edge weights between different expert clusters; Control the data size of a single expert cluster to not exceed the data threshold; The number of expert data units within a single expert cluster shall not exceed a certain threshold. Expert data units located at the same layer or at related layers are grouped into the same expert cluster.

6. The expert weight storage method according to claim 1, characterized in that, The storage of expert weights of the hybrid expert large language model in auxiliary memory based on the multiple expert clusters includes: Obtain the description information of each of the multiple expert clusters, and store the expert weights of the hybrid expert big language model in the auxiliary memory according to the description information of each of the multiple expert clusters; The description information of the expert cluster includes the identifier of the expert cluster, the identifier of each expert data unit in the expert cluster, and the data size of the expert cluster.

7. The expert weight storage method according to claim 6, characterized in that, The description information of the expert cluster also includes at least one of the following: the access popularity of the expert cluster, and the association strength metric of the expert cluster; The storage of expert weights of the hybrid expert large language model in auxiliary memory based on the multiple expert clusters includes: If the access popularity of the expert cluster is higher than the first popularity threshold, or if the association strength metric of the expert cluster is higher than the association strength threshold, the expert weight of the expert cluster is preferentially stored in a continuous page area or a continuous area within a block. If the access popularity of the expert cluster is lower than the second popularity threshold, the expert weights of the expert cluster are optimized for storage overhead and then stored in a physical region with a priority lower than the priority threshold; the second popularity threshold is less than the first popularity threshold.

8. The expert weight storage method according to claim 6, characterized in that, The description information of the expert cluster also includes the identifier of the target expert cluster that can be read in parallel with the expert cluster; The storage of expert weights of the hybrid expert large language model in auxiliary memory based on the multiple expert clusters includes: In the case where there is a target expert cluster that can be read in parallel, the expert weights of the expert clusters and the corresponding expert weights of the target expert clusters are stored in different channels or different chip units.

9. The expert weight storage method according to claim 6, characterized in that, After storing the expert weights of the hybrid expert large language model in the auxiliary memory based on the multiple expert clusters, the method further includes: Construct a layout metadata table; wherein, the layout metadata table includes the description information of the expert cluster and the physical layout information of the expert cluster in the auxiliary storage; The physical layout information of the expert cluster in the auxiliary memory includes the physical address range occupied by the expert data units of the expert cluster.

10. The expert weight storage method according to claim 9, characterized in that, The description information of the expert cluster also includes at least one of the following: updated version; storage overhead optimization information of the expert cluster; recommended prefetch window; prefetch priority; valid status identifier.

11. The expert weight storage method according to claim 9, characterized in that, The method further includes: Obtain the identifier of the prefetched expert data unit that is currently active or in the candidate state; Query the layout metadata table to obtain the prefetch expert cluster where the prefetch expert data unit is located, and obtain the physical layout information of the prefetch expert cluster in the auxiliary memory; Based on the physical layout information of the prefetched expert cluster in the auxiliary memory, a batch prefetch of expert weights is performed.

12. The expert weight storage method according to claim 1, characterized in that, The auxiliary storage includes NAND flash memory.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores at least one computer program that can be executed by the at least one processor, the at least one computer program being executed by the at least one processor to enable the at least one processor to perform the expert weight storage method as described in any one of claims 1-12.

14. A non-volatile computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, implements the expert weight storage method as described in any one of claims 1-12.

Images