Segmented expelling queue deepening multi-device hybrid expert model streaming service system and method

Abstract

The application relates to the field of artificial intelligence, in particular to a segmented expelling queue deepening multi-device hybrid expert model streaming service system and method, which comprises an expert weight fragmentation storage module, a segmented expelling cache module, a parallel missing issuance module, a pipeline ring module, a causal foresight prefetching module and a read event telemetry module. The segmented expelling cache module divides the explicit expert cache into a trial segment and a protection segment. By adopting the technical scheme, the beneficial effects of significantly improving the continuous read bandwidth and the end-to-end decoding speed can be achieved. In the case that the read missing times increase, the end-to-end decoding speed is still obviously improved. The joint design of the quantization bit width grading and the layer interval division enables the higher bit width model configuration to be fully cached and resident. The read event telemetry module provides read level observability for operation monitoring and simulation test bench calibration. The whole design does not depend on any predictor, training process or online statistical model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a segmented eviction queue-based multi-device hybrid expert model streaming service system and method. Background Technology

[0002] As the scale of language model parameters continues to grow, the weights of state-of-the-art open-weight hybrid expert models have reached hundreds of gigabytes or even terabytes, far exceeding the physical memory capacity of a single general-purpose computing device. Hybrid expert architectures set up hundreds of expert subnetworks at each decoding layer, with each token activating only a small number of experts. This sparse activation characteristic makes streaming service solutions a realistic way to run such massive models on ordinary hardware.

[0003] Under current technological conditions, expert weight caching management is a core bottleneck restricting the speed of streaming services. Existing technologies primarily rely on two approaches for expert caching management: one is eviction policy optimization aimed at improving cache hit rate, which involves introducing access frequency statistics and expert popularity profiling on top of the least recently used strategy, striving to retain the most frequently activated experts in memory; the other is a prefetching mechanism based on route prediction, which trains a predictor to predict which experts will be activated by future tokens and prefetches them. However, for modern hybrid expert models with highly distributed routing, both approaches fail: expert activation in such models is almost uniform, with the top 20% of experts only handling about 23% of the activations, resulting in minimal benefits from caching schemes based on popularity skew. Real-world testing shows that attention-based proxy route predictors achieve only about 21% recall on such models, and the experts read by prediction-driven prefetching are mostly not actually needed, instead crowding out cache and bandwidth. Furthermore, recent cache-aware routing schemes manipulate router logic values ​​during inference to favor cached experts. These schemes alter the model's actual routing decisions, posing a risk of output quality shift. Other training-time routing locality fine-tuning schemes retrain the model by adding a loss function to concentrate expert usage. These schemes require modification of the model weights themselves, making them unsuitable for deployment scenarios where the served model must maintain unchanged weights. The system described in this application neither alters any routing decisions nor requires any model retraining; the routing behavior and output of the served model remain consistent with the full-resident baseline.

[0004] More fundamentally, existing technologies implicitly assume that fewer misses result in faster speeds, treating the number of cache misses as the sole optimization objective. The inventors of this application, through read-level testing, discovered that in scenarios using solid-state storage (SSD) as the streaming medium, end-to-end latency is actually dominated by the arrival time structure of misses rather than the number of misses: the sustained read bandwidth of modern SSDs increases significantly with the depth of the concurrent command queue; the single-stream read bandwidth under a shallow queue is less than half the sustained bandwidth of a deep queue. If misses arrive in an isolated, intermittent manner, the SSD queue is repeatedly emptied, resulting in a longer total read time even with a small total number of misses. Existing caching technologies all use hit rate as the objective function; no eviction strategy aims to shape the arrival structure of misses or maintain a deep concurrent queue, nor does any mechanism sacrifice hit rate for higher sustained bandwidth. This results in streaming expert services being limited by shallow queue bandwidth, predictive failures, and single-machine capacity limits. There is an urgent need for a hybrid expert model streaming service solution that does not rely on route prediction and directly improves the effective bandwidth of SSDs through a deterministic caching mechanism. Summary of the Invention

[0005] The purpose of this invention is to provide a segmented eviction queue-based multi-device hybrid expert model streaming service system and method, which can effectively solve the technical defects of existing hybrid expert model streaming services that are limited by shallow queue bandwidth, route prediction failure and single-machine memory capacity limits.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A segmented eviction queue-based multi-device hybrid expert model streaming service system includes: The expert weight shard storage module is used to divide the decoding layer sequence of the hybrid expert model into N consecutive layer intervals and allocate them to N computing devices. Each computing device only persistently stores the weight shards of all experts in its allocated layer interval in its local non-volatile solid-state memory. The segmented eviction cache module is used to maintain an explicit expert cache with a capacity constrained by a preset byte budget in the unified memory of each computing device, and to divide the explicit expert cache into two sub-segments: a trial segment and a protected segment. Newly read expert weights are inserted into the trial segment and promoted to the protected segment when used again. The last bit of the protected segment is demoted back to the trial segment while retaining its weight, and the last bit of the trial segment is evicted. The parallel missing signature module is used to submit all the read requests of missing experts in each decoding layer to the asynchronous read thread pool at once before blocking when the layer is activated. The pipeline loop module is used to activate tensors with a data loop forward pass and maintain the number of flying tokens at a minimum of the pipeline stage N with an acknowledgment loop. The causal look-ahead prefetching module is used to prefetch the next-layer target expert using only the next-layer routing results generated by the current token, without setting any routing predictor; The read event telemetry module is used to record the issuance and completion timestamps of each read and persist them to the output.

[0007] Preferably, in the segmented eviction cache module, the proportion of the protection segment to the total capacity of the explicit expert cache is between 0.6 and 0.9, and the trial segment capacity ensures that the residence time of a newly inserted expert before being evicted is not less than 1.2 pipeline cycles, where one pipeline cycle is the time required for a token to complete the activation of all decoding layers under the jurisdiction of the device at this level.

[0008] Preferably, after the parallel missing sign-off module submits a missing read request all at once, the segmented eviction cache module receives the expert weights prefetched by the causal look-ahead prefetch module, and under the condition that the pipeline loop module maintains the number of flying tokens at no less than N, the solid-state memory of the same device can accept concurrent read requests from at least two decoding layers or at least two flying tokens at any given time.

[0009] Preferably, in the pipeline loop module, the acknowledgment loop maintains the number of fly tokens at the pipeline stage number N plus one; the data loop and the acknowledgment loop are implemented using Transmission Control Protocol streaming sockets.

[0010] Preferably, in the causal look-ahead prefetch module, the expert weights temporarily stored in the prefetch are not included in the segment structure of the segmented eviction cache module, and are only inserted into the trial segment when activated for use, with the temporary storage capacity subject to an independent upper limit constraint.

[0011] Preferably, the expert weight cache of each computing device consists of two levels: the first level is the segmented eviction cache with explicit prefetching, and the second level is the operating system page cache; the explicit cache byte budget is lower than the available physical memory of the device to reserve page cache space.

[0012] Preferably, the N computing devices are heterogeneous devices, and the length of each device layer interval is determined according to its memory capacity and the measured concurrent bandwidth limit of the solid-state storage.

[0013] Preferably, the expert weight shards are stored with a quantization width of four bits, six bits, or eight bits; in the four to eight bit width configuration, the total number of expert weights in the layer interval under the jurisdiction of the memory-constrained device does not exceed its explicit cache budget, so that the device achieves zero solid-state memory reads in steady state.

[0014] Preferably, the eviction decision of the segmented eviction cache module does not rely on any routing prediction, access frequency statistics model or trained reuse predictor, but is determined solely by the segment structure's insertion, promotion, demotion and eviction rules.

[0015] Another aspect of the present invention provides a method for streaming multi-device hybrid expert models, comprising: Step S1: Divide the decoding layer sequence of the hybrid expert model into N consecutive layer intervals and allocate them to N computing devices. Each computing device persistently stores the weight fragments of all experts in its allocated layer interval only in its local non-volatile solid-state memory. Step S2: Maintain an explicit expert cache with a capacity constrained by a preset byte budget in the unified memory of each computing device, and divide the explicit expert cache into two sub-segments: a trial segment and a protected segment; wherein, newly read expert weights are inserted into the trial segment, and when used again, they are promoted to the protected segment, the last bit of the protected segment is demoted back to the trial segment while retaining the weight, and the last bit of the trial segment is evicted; Step S3: At the start of activation of each decoding layer, submit all read requests for missing experts in that layer to the asynchronous read thread pool at once before blocking. Step S4: Activate the tensor with a data loop forward pass and maintain it with an acknowledgment loop while the number of flying tokens is not less than the pipeline stage number N; Step S5: Prefetch the next-layer target expert using only the next-layer routing results generated by the current token, without setting any routing predictor; Step S6: Record the issuance and completion timestamps for each read and persist them to the output.

[0016] Compared with the prior art, the present invention has the following beneficial effects: By using a segmented eviction cache module, cache misses are clustered into a series of continuously overlapping read requests based on purely deterministic segment structure rules. This directly deepens the concurrent command queue of the solid-state memory. In actual tests, the continuous read bandwidth increased from 3.54 gigabytes per second to 5.31 gigabytes per second, an improvement of about 50%. With the number of read misses increasing by about 24%, the end-to-end decoding speed was improved by 27%-28% compared with the same feature set. This establishes "trading bandwidth for missing structure" as a new optimization dimension orthogonal to "reducing misses", without the need for any predictors, training processes or online statistical models. By coordinating parallel missing signatures, causal look-ahead prefetching, and the lower limit of in-flight depth, read requests at each stage of the pipeline continuously overlap. The measured time-weighted average queue depth increased from 5.10 to 5.92, and the proportion of input / output time with a depth of not less than six increased from 40.9% to 52.7%. The 24% increase in read requests resulted in a 9% reduction in device busy time. By connecting multiple general-purpose devices via a pipelined loop module using a common transmission control protocol network, usable speed services for ultra-large hybrid expert models that are impossible to support with the memory capacity of a single machine are achieved. In actual tests, the speed of a 372-gigabyte model was increased from 0.58 tokens per second to 2.76 tokens per second on 2-3 96-gigabyte models, which is about 4.7 times. The long-term operation retention rate was 92.7%, and this mechanism was reproduced on the third-level topology of a cross-border wide area network. By combining the design of quantization bit width gradation and layer interval division, higher bit width and higher quality models can be configured to run on the same device cluster in a way that allows memory-constrained devices to reside in full cache, thus simultaneously scaling up quality and capacity. The read event telemetry module provides read-level observability, and the queue depth profile is used for both operation monitoring and calibration of a discrete event simulation test bench that can reproduce all measured points, so that any subsequent cache or scheduling changes can be predicted in the simulation at a very low cost. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall technical architecture of the system proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework of the segmented eviction cache module in this invention; Figure 3 This is a logical flow diagram of the parallel collaborative operation of multiple devices in this invention; Figure 4 This is a schematic diagram comparing the decoding throughput of the present invention with that of the prior art; Figure 5 This is a schematic diagram of the overall system structure in this invention; Figure 6 This is a comparison chart of the final measured queue depth occupancy distribution in this invention; Figure 7 This is a comparison chart of the final measured reading interval timelines in this invention; Figure 8 This is a schematic diagram of the final segmented eviction cache segment structure and state transition in this invention; Figure 9 This is a schematic diagram illustrating the relationship between the final pipeline loop's on-flight depth and decoding speed in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the accompanying drawings, the size of constituent elements, the thickness of layers, or areas may sometimes be exaggerated for clarity. Therefore, any implementation of this disclosure is not necessarily limited to the dimensions shown in the drawings, and the shapes and sizes of the components in the drawings do not reflect true proportions. Furthermore, the drawings schematically illustrate ideal examples, and any implementation of this disclosure is not limited to the shapes or values ​​shown in the drawings.

[0020] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a segmented eviction queue-based multi-device hybrid expert model streaming service system proposed in one embodiment of this application. As shown in the figure, the system includes: an expert weight shard storage module, a segmented eviction cache module, a parallel missing item issuance module, a pipeline loop module, a causal lookahead prefetching module, and a read event telemetry module.

[0021] In the hybrid expert model streaming service system, the front-end device divides the decoding layer sequence of the hybrid expert model into N consecutive layer intervals and distributes them to N computing devices through the expert weight sharding storage module. Each computing device persistently stores the weight shards of all experts in its assigned layer interval only in its local non-volatile solid-state memory. The weight tensors of each expert are arranged continuously along the first axis in the storage layout, so that a single expert can be read in as a whole in a single pre-read operation without rearrangement.

[0022] In this embodiment, the hybrid expert model streaming service system manages the weights of the hybrid expert model through an expert weight sharding storage module. Specifically, the complete hybrid expert model is divided into N consecutive layer intervals according to the order of the decoding layers, and each layer interval is assigned to a computing device for processing. For example, when the hybrid expert model contains eighty decoder layers, it can be divided into four consecutive layer intervals, each containing twenty decoder layers, and assigned to four computing devices respectively. Each computing device persistently stores all expert weights within its assigned layer interval in local non-volatile solid-state memory. In terms of storage layout design, the weight tensors of each expert are stored in a continuous arrangement along the first axis, which is the primary dimension of the weight tensor, usually the input dimension or the output dimension. This storage layout ensures that when reading a single expert weight, only a pre-read operation is needed to read the entire expert weight into unified memory, without performing multiple discrete reads or weight rearrangement operations, significantly reducing read latency and improving memory access efficiency.

[0023] The segmented eviction cache module maintains an explicit expert cache in the unified memory of each computing device, with a capacity constrained by a preset byte budget. This explicit expert cache is divided into two sub-segments: a trial segment and a protected segment. Newly read expert weights are inserted into the trial segment. When used again, they are promoted to the protected segment. The last expert in the protected segment is demoted back to the trial segment while retaining its weight, and the last expert in the trial segment is evicted. The capacity ratio of the trial segment and the protected segment ensures that the residency time of newly inserted experts in the trial segment before being evicted is no less than a preset pipeline cycle multiple. This ensures that experts reused across tokens stably reside in the protected segment, and missing experts are concentrated in the trial segment.

[0024] In this embodiment, the segmented eviction cache module is the core innovative component of this invention. Its design goal is to cluster cache misses into a continuous series of overlapping read requests through deterministic segment structure rules, thereby deepening the concurrent command queue of the solid-state memory and improving continuous read bandwidth. Figure 2 As shown, the segmented eviction cache module divides the explicit expert cache into two sub-segments: a trial segment and a protection segment. The trial segment is used to hold newly added expert weights, while the protection segment protects frequently accessed expert weights. When a new expert weight is read in, it is first inserted at the end of the trial segment; when the expert weight is used again, it is promoted from the trial segment to the protection segment; when the protection segment is full, the expert weight at the end of the protection segment is demoted back to the trial segment; when the trial segment is full, the expert weight at the end of the trial segment is evicted to solid-state memory.

[0025] In this embodiment, the proportion of the protected segment to the total capacity of the explicit expert cache is between 0.6 and 0.9, preferably 0.8. When the proportion reaches 0.9 or higher, the promotion channel almost stops, and the effective cache capacity shrinks sharply. When the proportion of the protected segment is 0.8, the proportion of the trial segment is 0.2. At this time, newly inserted experts can reside in the trial segment for a sufficiently long time before being evicted, ensuring that experts who are reused across tokens reside stably in the protected segment, while missing requests are concentrated on the flowing members in the trial segment.

[0026] In this embodiment, the trial segment capacity ensures that the residency time of a newly inserted expert before being evicted is no less than 1.2 pipeline cycles. One pipeline cycle is the time required for a token to complete the activation of all decoding layers under the jurisdiction of this level of device. Assuming a pipeline cycle is T seconds, the trial segment capacity design ensures that the newly inserted expert weight resides in the cache for at least 1.2T seconds, during which at least one or more token requests can be processed. This design allows expert weights reused across tokens to reside stably in the protection segment during steady-state decoding, while newly requested expert weights flow in the trial segment. When the expert weight at the end of the trial segment is evicted, it has completed at least one pipeline cycle.

[0027] In this embodiment, the segmented eviction cache module, the parallel missing issuance module, the pipeline loop module, and the causal look-ahead prefetch module work together in operation, so that during steady-state decoding, the solid-state memory of the same device can accept concurrent read requests from at least two decoding layers or at least two flying tokens that overlap in time at any given moment. This clusters cache misses into a series of continuously overlapping read requests, deepens the concurrent command queue of the solid-state memory, and increases the continuous read bandwidth with the concurrency depth.

[0028] At the start of each decoding layer activation, the parallel missing signature module submits all the read requests for missing experts in that layer to the asynchronous read thread pool at once before blocking.

[0029] In this embodiment, in a traditional hybrid expert model streaming service system, expert weight read requests are typically issued sequentially, meaning only one read request for a missing expert is submitted at a time, and the next request is submitted only after that request is completed. This sequential issuance method results in insufficient depth of the concurrent command queue of the solid-state storage (SSD), failing to fully utilize the parallel read capability of the SSD. This invention addresses this by using a parallel missing expert issuance module. At the start of each decoding layer activation, all read requests for missing experts in that layer are immediately submitted to an asynchronous read thread pool. After receiving all requests, the asynchronous read thread pool schedules multiple asynchronous read threads to simultaneously initiate read requests to the SSD, thereby forming a concurrent read request queue on the SSD side and increasing the depth of the concurrent command queue.

[0030] The pipeline loop module activates the tensor by forwarding the data loop and maintains it with the acknowledgment loop at a number of flying tokens not less than the pipeline stage number N.

[0031] In this embodiment, the pipeline loop module is a key component for realizing multi-device collaborative streaming services. It comprises two loop structures: a data loop and an acknowledgment loop. The data loop is responsible for forwarding the activation tensor. That is, after a token completes its activation calculation at the current layer on one device, its activation tensor is passed to the next device through the data loop for further processing. The acknowledgment loop is responsible for maintaining the number of tokens in flight. That is, after the activation tensor of a token is passed to the next device, the next device sends an acknowledgment signal to the current device. Only after receiving the acknowledgment signal does the current device release the resources corresponding to the token, thus ensuring that the number of tokens in flight is always maintained above the pipeline stage number N.

[0032] In this embodiment, the acknowledgment ring maintains the number of in-fly tokens at the pipeline stage number N plus one. When the number of in-fly tokens falls below N, the end-to-end decoding speed drops sharply. This is because there are dependencies between the stages of the pipeline ring. If the number of in-fly tokens is insufficient at any given time, it will cause idle periods in the pipeline, and subsequent devices will need to wait for the upstream device to send the activation tensor, resulting in a decrease in overall throughput. By maintaining the number of in-fly tokens at N plus one in the acknowledgment ring, it is ensured that each stage of the pipeline ring always has enough in-fly tokens, allowing the data ring to continuously transmit the activation tensor and preventing pipeline idle periods.

[0033] In this embodiment, the data ring and acknowledgment ring are implemented using Transmission Control Protocol (TCP) streaming sockets, operating in ordinary local area networks (LANs) or cross-border wide area networks (WANs) without requiring dedicated high-speed interconnects. The activation tensor is transmitted at a granular level, once per token per level. This implementation allows the invention to be deployed in general network environments without relying on expensive high-speed interconnect hardware. Through TCP streaming sockets, the data ring and acknowledgment ring directly transmit the activation tensor and acknowledgment signal as a byte stream, resulting in low protocol overhead and high transmission efficiency.

[0034] In another embodiment, the data ring and the acknowledgment ring operate on a high-speed point-to-point interconnect (e.g., lightning bridging or remote direct memory access network), further reducing inter-level transmission latency; the streaming socket implementation and the high-speed interconnect implementation can be used interchangeably between different device pairs in the same system.

[0035] The causal look-ahead prefetching module only uses the next-layer routing results generated by the current token to prefetch the next-layer target expert, without setting any routing predictor.

[0036] In this embodiment, the causal look-ahead prefetching module is designed to prefetch the target expert weights for the next layer using the routing results of the current token, without relying on any form of route predictor. Specifically, in each layer of the hybrid expert model, the router determines which experts to send the current token to for processing based on the input activation tensor. After the current token is activated in the current layer, the causal look-ahead prefetching module immediately obtains the routing results of the token in the next layer and initiates a read request for the target expert weights of the next layer from the solid-state storage in advance based on the routing results. Since the routing results are actually generated by the current layer of the current token and are not prediction results, there is no risk of prediction failure, ensuring the accuracy of prefetching.

[0037] In this embodiment, the expert weights prefetched and temporarily stored are not included in the segment structure. They are only inserted into the trial segment when activated and used, and the storage capacity is subject to an independent upper limit. The expert weights prefetched from the solid-state memory by the prefetch module are stored in an independent prefetch buffer, which does not participate in the segment structure management of the segment eviction cache module. When an expert weight in the prefetch buffer is actually used, that expert weight is inserted from the prefetch buffer into the trial segment of the segment eviction cache module. The prefetch buffer has an independent capacity upper limit. When the capacity of the prefetch buffer reaches the upper limit, new prefetch requests will be rejected or delayed.

[0038] The event telemetry module records the issuance and completion timestamps for each read and persists them.

[0039] In this embodiment, the read event telemetry module provides fine-grained observability of read performance. The issuance and completion times of each read request are recorded and saved to the storage system via a persistent output mechanism. This timestamp data can be used to calculate the empirical distribution of time-weighted queue depth occupancy, the proportion of deep queue input / output time, and the duration of a single read service relative to concurrency depth. This empirical distribution is used to calibrate the discrete event simulation testbed, allowing any subsequent cache or scheduling changes to be predicted in the simulation at extremely low cost.

[0040] In this embodiment, the timestamp data recorded by the read event telemetry module includes: a unique identifier for the read request, the timestamp of the read request issuance, the timestamp of the read request completion, the expert identifier corresponding to the read request, the device identifier corresponding to the read request, and the data size of the read request. By analyzing this data, the service duration, queue waiting time, and read performance distribution under different concurrency depths can be accurately calculated for each read request.

[0041] In one embodiment, this application also provides a segmented eviction queue-based multi-device hybrid expert model streaming service system. In this system, the expert weight cache of each computing device constitutes a two-level hierarchy: the first level is an explicitly pre-read segmented eviction cache, and the second level is an operating system page cache. The explicit cache byte budget is lower than the available physical memory of the device to reserve page cache space.

[0042] In this embodiment, the expert weight cache of each computing device adopts a two-level hierarchical structure. The first level is an explicitly pre-fetched segmented eviction cache, which is directly managed by the segmented eviction cache module, and its capacity is determined by a preset byte budget. The second level is the operating system page cache. When expert weights are read from solid-state storage, they first enter the segmented eviction cache. If expert weights in the segmented eviction cache are evicted, these evicted expert weights do not immediately disappear from memory, but are managed by the operating system and may reside in the page cache. By preserving the viability of the page cache, even if expert weights in the explicit cache are evicted, the operating system may still hit these weights through the page cache, thereby providing additional cache hit opportunities.

[0043] In one embodiment, this application also provides a segmented eviction queue-based multi-device hybrid expert model streaming service system, in which N computing devices are heterogeneous devices, and the length of each device layer interval is determined according to its memory capacity and the measured concurrent bandwidth limit of solid-state storage.

[0044] In this embodiment, the N computing devices can be heterogeneous devices of different specifications, such as unified memory with different capacities and solid-state storage with different performance. The layer interval length of each device is dynamically determined based on its memory capacity and the measured concurrent bandwidth limit of the solid-state storage. Devices with larger memory capacities can be allocated more layer intervals, and devices with higher solid-state storage bandwidth can be allocated more layer intervals. Through this adaptive layer interval allocation strategy, the hardware capabilities of each device can be maximized, achieving overall system load balancing.

[0045] In one embodiment, this application also provides a segmented eviction queue-based multi-device hybrid expert model streaming service system, in which expert weights are stored in 4-bit, 6-bit, or 8-bit quantization widths. With a higher bit width configuration, by correspondingly shortening the length of the layer interval under the device's jurisdiction, the total number of expert weights in the layer interval under the memory-constrained device does not exceed its explicit cache budget, enabling the device to achieve zero solid-state memory reads in steady state.

[0046] In this embodiment, expert weight slices can be stored using different quantization bit widths, including four-bit quantization, six-bit quantization, or eight-bit quantization. A lower quantization bit width results in less storage space but a larger quantization error. With a higher bit width configuration, by correspondingly shortening the length of the layer interval under the device's jurisdiction, the total amount of expert weights in the layer interval under the memory-constrained device is strictly controlled within the explicit cache budget. This ensures that the device does not need to read any expert weights from the solid-state memory during steady-state decoding; all expert weights reside in the segmented eviction cache, thereby completely eliminating solid-state memory read latency and achieving zero solid-state memory reads.

[0047] In one embodiment, this application also provides a segmented eviction queue-based multi-device hybrid expert model streaming service system, in which eviction decisions do not rely on any routing prediction, access frequency statistics model, or trained reuse predictor, but are generated solely by the deterministic rules of segment structure insertion, promotion, demotion, and eviction.

[0048] In another implementation, the hybrid expert model being served is a derived model pre-tuned with centralized objectives in the working set and with its weights frozen before deployment. This tuning is completed before deployment, the model weights remain unchanged during service, and the system does not alter any of its routing decisions, thus orthogonally superimposed on the system's deterministic caching mechanism. The applicant's cache simulation tests (least recently used strategy, budget of 64 experts) demonstrate that this type of tuning can reduce the expert missing rate per sequence from approximately 64% to approximately 18%, further improving streaming service throughput with the same caching budget.

[0049] In this embodiment, the eviction decision of the segmented eviction cache module is generated entirely by deterministic rules, without relying on any statistical or learning-based prediction models. When a new expert weight is inserted into the trial segment, the expert weight at the end of the trial segment is evicted according to the first-in, first-out principle; when an expert weight is reused, it is promoted from its current position to the protected segment; when the protected segment is full, the expert weight at the end of the protected segment is demoted to the trial segment. This deterministic rule ensures the high predictability and repeatability of caching behavior, avoiding the prediction failure risk that may arise from prediction models.

[0050] In one embodiment, this application also provides a multi-device hybrid expert model streaming service method, the method comprising: Step S1: Divide the decoding layer sequence of the hybrid expert model into N consecutive layer intervals and distribute them to N computing devices. Each computing device persistently stores the weight fragments of all experts in its assigned layer interval only in its local non-volatile solid-state memory.

[0051] In this embodiment, step S1 implements the fragmented storage of the hybrid expert model. All decoding layers of the hybrid expert model are sequentially divided into N consecutive layer intervals, each containing several consecutive decoding layers. Each layer interval is assigned to a computing device, which is responsible for processing the activation calculations of the decoding layers falling within its layer interval. Each computing device only stores the expert weights of its assigned layer interval in its local solid-state memory, without needing to access the storage resources of other devices, thereby minimizing cross-device communication overhead.

[0052] Step S2: Maintain an explicit expert cache in the unified memory of each computing device, with a capacity constrained by a preset byte budget, and divide the explicit expert cache into two sub-segments: a trial segment and a protected segment. Newly read expert weights are inserted into the trial segment. When used again, they are promoted to the protected segment. The last bit in the protected segment is demoted back to the trial segment while retaining its weight, and the last bit in the trial segment is evicted.

[0053] In this embodiment, step S2 implements a segmented eviction cache management strategy. Each computing device allocates a portion of its unified memory as an explicit expert cache, the capacity of which is determined by a preset byte budget. The explicit expert cache is divided into two sub-segments: a trial segment and a protected segment, with a capacity ratio of 0.8 for the protected segment and 0.2 for the trial segment. When a new expert weight is read from the solid-state storage into the unified memory, the expert weight is inserted at the end of the trial segment; when the expert weight is used again, it is promoted from the trial segment to the protected segment; when the protected segment is full, the expert weight at the end of the protected segment is demoted back to the trial segment; when the trial segment is full, the expert weight at the end of the trial segment is evicted to the solid-state storage.

[0054] Step S3: At the start of each decoding layer activation, submit all reading requests for missing experts in that layer to the asynchronous reading thread pool all at once before blocking.

[0055] In this embodiment, step S3 implements parallel missing signature. When a decoding layer begins activation computation, it first checks whether all expert weights required by that layer already exist in the segmented eviction cache. For expert weights not in the cache, i.e., experts with cached missing weights, all read requests for missing experts are submitted to the asynchronous read thread pool at once. The asynchronous read thread pool schedules multiple read threads to simultaneously initiate read requests to the solid-state storage, thus achieving parallelization of read requests.

[0056] Step S4: Activate the tensor with a data loop forward pass and maintain it with an acknowledgment loop while the number of flying tokens is not less than the pipeline stage number N.

[0057] In this embodiment, step S4 implements data flow in the pipeline loop. After a token completes its layer interval activation calculation on a device, the resulting activation tensor is passed to the next device via the data loop for further processing. The data loop is implemented using Transmission Control Protocol (TCP) streaming sockets and can operate on ordinary local area networks (LANs) or cross-border wide area networks (WANs). The acknowledgment loop maintains the number of tokens in flight. When a downstream device completes its activation calculation, it sends an acknowledgment signal to the upstream device. Upon receiving the acknowledgment signal, the upstream device releases the resources corresponding to the token. The acknowledgment loop ensures that the number of tokens in flight is always maintained at pipeline stage N plus one, avoiding idle periods in the pipeline.

[0058] Step S5: Prefetch the next-layer target expert using only the next-layer routing results generated by the current token, without setting any routing predictor.

[0059] In this embodiment, step S5 implements causal look-ahead prefetching. After the current layer of the current token is activated, the causal look-ahead prefetching module obtains the routing result of the token in the next layer. This routing result is actually generated by the current layer of the current token, not a predicted result. Based on the routing result of the next layer, the causal look-ahead prefetching module initiates a prefetching request for the target expert weight of the next layer to the solid-state storage. Since the routing result is deterministic, the accuracy of prefetching is guaranteed.

[0060] Step S6: Record the issuance and completion timestamps for each read and persist them to the output.

[0061] In this embodiment, step S6 implements read event telemetry. The issuance and completion times of each read request are recorded and persistently output to the storage system along with the read request's metadata. This data can be used for subsequent performance analysis and simulation calibration.

[0062] In step S2, the protection segment accounts for 0.8% of the total capacity of the explicit expert cache, and the trial segment capacity ensures that the residence time of a newly inserted expert before being evicted is not less than 1.2 pipeline cycles.

[0063] In this embodiment, when the protection segment ratio is 0.8, the trial segment ratio is 0.2. Assuming the pipeline cycle is T, the capacity design of the trial segment ensures that newly inserted expert weights reside in the trial segment for at least 1.2T seconds before being evicted, guaranteeing that the expert weight is used at least once.

[0064] In step S4, it is confirmed that the number of flying tokens will be maintained at pipeline stage N plus one.

[0065] In this embodiment, the acknowledgment loop maintains the number of in-flight tokens by receiving acknowledgment signals from downstream devices. When the number of in-flight tokens falls below N, idle periods occur between stages of the pipeline loop, causing a sharp drop in end-to-end decoding speed. By maintaining the number of in-flight tokens at N plus one, the pipeline is ensured to always be at full capacity.

[0066] In step S1, the length of each device layer interval is determined based on its memory capacity and the measured concurrent bandwidth limit of the solid-state memory. Expert weight fragments are stored with a quantization bit width of four bits, six bits, or eight bits.

[0067] In this embodiment, the layer interval length of each device is dynamically adjusted according to its hardware capabilities. Devices with larger memory capacity can be allocated more layer intervals, as can devices with higher solid-state storage bandwidth. Expert weight sharding can use different quantization bit widths, and the choice of quantization bit width depends on the balance between memory capacity and performance requirements.

[0068] In step S6, the read-level timestamp is used to calculate the time-weighted queue depth occupancy distribution, the deep queue input / output time ratio, and the empirical distribution of single read service duration on concurrency depth.

[0069] In this embodiment, the timestamp data recorded by the read event telemetry module can be used to calculate various performance metrics. The time-weighted queue depth occupancy distribution reflects the depth distribution of the solid-state memory concurrent command queue during the decoding process. The deep queue input / output time ratio reflects the proportion of input / output operations in the total time under high concurrency depth. The empirical distribution of single read service duration versus concurrency depth reflects the service duration distribution of each read request under different concurrency depths. These empirical distributions can be used to calibrate discrete event simulation test benches, enabling subsequent system optimizations to be verified at low cost in a simulation environment.

[0070] Reference Figure 3 , Figure 3 This is a logical flow diagram illustrating the collaborative operation of multi-device parallel missing document issuance and pipeline loop in one embodiment of this application. In the multi-device hybrid expert model streaming service system, the collaborative operation of data flow and control flow follows the process described below.

[0071] At time step t, token A on device one enters its layer interval and begins decoding layer activation calculation. Before the activation calculation begins, the parallel missing issuance module checks whether all expert weights required for decoding layers within the layer interval under device one's jurisdiction already exist in the segmented eviction cache. For expert weights not in the cache, the parallel missing issuance module submits all read requests to the asynchronous read thread pool at once. The asynchronous read thread pool schedules multiple read threads to simultaneously read the missing expert weights from the solid-state storage.

[0072] Before the activation tensor of token A is passed from device one to device two, the data ring first checks the segmented eviction cache status on device two. If the expert weight required for the next layer of token A exists in the cache of device two, the activation tensor can directly enter device two for the next layer calculation. If the required expert weight does not exist in the cache of device two, the parallel missing issuance module is triggered, and the read request for the missing expert is submitted to the asynchronous read thread pool.

[0073] After token A completes the activation calculation at each level, the acknowledgment ring sends an acknowledgment signal to the upstream device. Upon receiving the acknowledgment signal, the upstream device releases the cache space and computing resources corresponding to that token and allows new tokens to enter the pipeline. Through the coordination of the acknowledgment ring, the number of tokens in flight is always maintained at pipeline stage N plus one, ensuring the continuous operation of the pipeline ring.

[0074] After token A is activated at the current layer, the causal look-ahead prefetch module uses the routing results generated at the current layer to prefetch the target expert weights for token A at the next layer. The prefetched expert weights are stored in a separate prefetch buffer. When token A enters the next layer, the expert weights in the prefetch buffer are inserted into the trial segment of the segmented eviction cache.

[0075] The read event telemetry module continuously records the issuance and completion times of each read request throughout the entire process and persists these timestamp data for subsequent performance analysis and simulation calibration.

[0076] The segmented eviction queue-based multi-device hybrid expert model streaming service system can be deployed in a multi-level architecture of terminal devices and cloud servers.

[0077] The terminal device is responsible for receiving the request token input by the user and sending it to the cloud server. After receiving the request token, the cloud server distributes it to the multi-device hybrid expert model streaming service system for processing. The multi-device hybrid expert model streaming service system consists of N computing devices, each responsible for processing one layer interval of the hybrid expert model. The devices communicate with each other via data loops and acknowledgment loops to transfer activation tensors and synchronize token states.

[0078] After the request token completes the activation calculation of all decoding layers in the multi-device hybrid expert model streaming service system, the resulting output is returned to the terminal device via the cloud server. The terminal device receives the output and performs subsequent processing.

[0079] During the multi-level interaction process, the read event telemetry module continuously records the read performance data on each device and sends the timestamp data to the monitoring server via the network. The monitoring server analyzes the timestamp data and generates a performance report for system maintenance personnel to refer to.

[0080] Reference Figure 4 , Figure 4 This diagram illustrates a comparison of the core principle of this application with existing technologies in terms of decoding throughput. By deepening the core technology of the multi-device hybrid expert model streaming service system through segmented eviction queues, the system achieves a significant improvement in decoding throughput.

[0081] In existing hybrid expert model streaming service systems, single-device decoding speed is limited due to shallow queue bandwidth constraints and route prediction failures. By employing deterministic segment structure rules in the segmented eviction cache module, cache misses are clustered into continuously overlapping read request sequences, deepening the concurrent command queue of the solid-state storage. In actual testing, sustained read bandwidth increased from 3.54 gigabytes per second to 5.31 gigabytes per second, an improvement of approximately 50%.

[0082] Through the collaborative work of the parallel missing signature module, the causal lookahead prefetch module, and the pipeline loop module, read requests at each stage of the pipeline continuously overlap, increasing the time-weighted average queue depth from 5.10 to 5.92, and raising the proportion of input / output time with a depth of at least six from 40.9% to 52.7%. Despite an approximately 24% increase in the number of read misses, the end-to-end decoding speed improved by 27% to 28%.

[0083] Through a pipelined loop module, multiple general-purpose devices connected via a standard transmission control protocol network achieve usable speed services for ultra-large hybrid expert models that are impossible for a single machine's memory capacity to handle. In actual testing, a 372-gigabyte model saw its speed increased from 0.58 tokens per second to 2.76 tokens per second on two to three 96-gigabyte models, a 4.7-fold increase, with a long-term maintenance rate of 92.7%.

[0084] The segmented eviction queue-based multi-device hybrid expert model streaming service system proposed in this application has the following significant advantages: 1. By using a segmented eviction cache module to cluster cache misses into a series of continuously overlapping read requests based on purely deterministic segment structure rules, the concurrent command queue of the solid-state memory is directly deepened. In actual tests, the continuous read bandwidth increased from 3.54 gigabytes per second to 5.31 gigabytes per second, an increase of about 50%. With the number of read misses increasing by about 24%, the end-to-end decoding speed increased by 27% to 28% compared with the same feature set. This establishes "trading bandwidth for missing structure" as a new optimization dimension orthogonal to "reducing misses", without the need for any predictors, training processes or online statistical models.

[0085] Second, by coordinating parallel missing signature, causal look-ahead prefetching, and the lower limit of in-flight depth, read requests at each stage of the pipeline continuously overlap. The measured time-weighted average queue depth increased from 5.10 to 5.92, and the proportion of input / output time with a depth of not less than six increased from 40.9% to 52.7%. The 24% increase in read requests resulted in a 9% reduction in device busy time.

[0086] Third, by connecting multiple general-purpose devices via a pipelined loop module using a common transmission control protocol network, the available speed service for ultra-large hybrid expert models that cannot be supported by the memory capacity of a single machine is achieved. In actual tests, the speed of a 372-gigabyte model increased from 0.58 tokens per second to 2.76 tokens per second on two to three 96-gigabyte models, an increase of about 4.7 times. The long-term operation retention rate was 92.7%, and this mechanism was reproduced on the third-level topology of a cross-border wide area network.

[0087] Fourth, through the joint design of quantization bit width hierarchy and layer interval division, higher bit width and higher quality models are configured to run on the same device cluster in a way that allows memory-constrained devices to reside in full cache, thus simultaneously scaling up quality and capacity.

[0088] Fifth, the read event telemetry module provides read-level observability. The queue depth profile is used for both operation monitoring and calibration of a discrete event simulation test bench that can reproduce all measured points, so that any subsequent cache or scheduling changes can be predicted in the simulation at a very low cost.

[0089] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0090] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0095] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0096] The above provides a detailed description of the segmented eviction queue-based multi-device hybrid expert model streaming service system and method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A segmented eviction queue-based multi-device hybrid expert model streaming service system, characterized in that, include: The expert weight shard storage module is used to divide the decoding layer sequence of the hybrid expert model into N consecutive layer intervals and allocate them to N computing devices. Each computing device only persistently stores the weight shards of all experts in its allocated layer interval in its local non-volatile solid-state memory. The segmented eviction cache module is used to maintain an explicit expert cache with a capacity constrained by a preset byte budget in the unified memory of each computing device, and to divide the explicit expert cache into two sub-segments: a trial segment and a protected segment. Newly read expert weights are inserted into the trial segment and promoted to the protected segment when used again. The last bit of the protected segment is demoted back to the trial segment while retaining its weight, and the last bit of the trial segment is evicted. The parallel missing signature module is used to submit all the read requests of missing experts in each decoding layer to the asynchronous read thread pool at once before blocking when the layer is activated. The pipeline loop module is used to activate tensors with a data loop forward pass and maintain the number of flying tokens at a minimum of the pipeline stage N with an acknowledgment loop. The causal look-ahead prefetch module is used to prefetch the next-layer target expert using only the next-layer routing results generated by the current token, without setting any routing predictor.

2. The segmented eviction queue-based multi-device hybrid expert model streaming service system according to claim 1, characterized in that, In the segmented eviction cache module, the proportion of the protected segment to the total capacity of the explicit expert cache is between 0.6 and 0.9, and the capacity of the trial segment ensures that the residence time of a newly inserted expert before being evicted is not less than 1.2 pipeline cycles. One pipeline cycle is the time required for a token to complete the activation of all decoding layers under the jurisdiction of the device at this level.

3. The segmented eviction queue-based multi-device hybrid expert model streaming service system according to claim 1, characterized in that, After the parallel missing token issuance module submits a missing read request all at once, the segmented eviction cache module receives the expert weights prefetched by the causal look-ahead prefetch module, and under the condition that the pipeline loop module maintains the number of flying tokens at no less than N, the solid-state memory of the same device can accept concurrent read requests from at least two decoding layers or at least two flying tokens at any given time.

4. The segmented eviction queue-based multi-device hybrid expert model streaming service system according to claim 1, characterized in that, In the pipeline loop module, the acknowledgment loop maintains the number of fly tokens at the pipeline stage number N plus one; the data loop and the acknowledgment loop are implemented using Transmission Control Protocol streaming sockets.

5. The segmented eviction queue-based multi-device hybrid expert model streaming service system according to claim 1, characterized in that, In the causal look-ahead prefetch module, the expert weights temporarily stored in the prefetch are not included in the segment structure of the segmented eviction cache module. They are only inserted into the trial segment when they are activated and used, and the temporary storage capacity is subject to an independent upper limit constraint.

6. The segmented eviction queue-based multi-device hybrid expert model streaming service system according to claim 1, characterized in that, The expert weight cache for each of the aforementioned computing devices consists of two levels: the first level is the segmented eviction cache with explicit prefetching, and the second level is the operating system page cache; the explicit cache byte budget is lower than the available physical memory of the device to reserve space for the page cache.

7. A segmented eviction queue-based multi-device hybrid expert model streaming service system according to claim 1, characterized in that, The N computing devices are heterogeneous devices, and the length of each device layer interval is determined according to its memory capacity and the measured concurrent bandwidth limit of the solid-state storage.

8. A segmented eviction queue-based multi-device hybrid expert model streaming service system according to claim 1, characterized in that, The expert weight slices are stored with a quantization bit width of four bits, six bits, or eight bits. With a four- to eight-bit width configuration, the total number of expert weights in the layer range of a memory-constrained device does not exceed its explicit cache budget, enabling the device to achieve zero solid-state memory reads in steady state.

9. A segmented eviction queue-based multi-device hybrid expert model streaming service system according to claim 1, characterized in that, The eviction decision of the segmented eviction cache module does not rely on any routing prediction, access frequency statistics model or trained reuse predictor, but is determined solely by the segment structure's insertion, promotion, demotion and eviction rules.

10. A segmented eviction queue-based multi-device hybrid expert model streaming service system according to claim 1, characterized in that, It also includes a read event telemetry module, which records the issuance and completion timestamps of each read and persists them.

11. A segmented eviction queue-based multi-device hybrid expert model streaming service system according to claim 1, characterized in that, The hybrid expert model is a derivative model that has been pre-tuned in a centralized working set with the optimization objectives of the number of effective experts per sequence window and the cache miss rate, and whose weights are frozen before deployment. During the service period, the weights of the model remain unchanged, and the system does not change any of its routing decisions. The steady-state miss rate of the segmented eviction cache module is reduced accordingly.

12. A multi-device hybrid expert model streaming service method, characterized in that, include: Step S1: Divide the decoding layer sequence of the hybrid expert model into N consecutive layer intervals and allocate them to N computing devices. Each computing device persistently stores the weight fragments of all experts in its allocated layer interval only in its local non-volatile solid-state memory. Step S2: Maintain an explicit expert cache with a capacity constrained by a preset byte budget in the unified memory of each computing device, and divide the explicit expert cache into two sub-segments: a trial segment and a protected segment; wherein, newly read expert weights are inserted into the trial segment, and when used again, they are promoted to the protected segment, the last bit of the protected segment is demoted back to the trial segment while retaining the weight, and the last bit of the trial segment is evicted; Step S3: At the start of activation of each decoding layer, submit all read requests for missing experts in that layer to the asynchronous read thread pool at once before blocking. Step S4: Activate the tensor with a data loop forward pass and maintain it with an acknowledgment loop while the number of flying tokens is not less than the pipeline stage number N; Step S5: Prefetch the next-layer target expert using only the next-layer routing results generated by the current token, without setting any routing predictor.

Images