Storage scheduling method and system for large model decoding stage
By dynamically adjusting the number of candidate overlapping layers using a scheduler, combined with transport delay correction and marginal benefit density, the utilization of storage resources in the large model decoding stage is optimized, solving the problems of resource waste and strain, and improving resource utilization efficiency.
Patent Information
- Application Number
- CN202611281889.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-21
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, the storage scheduling method of large models during the decoding stage cannot be dynamically adjusted, resulting in resource waste in short context scenarios and resource shortage in long context scenarios. Furthermore, the changes in the transfer tasks between successive rounds during continuous decoding are not effectively managed, causing a mismatch between the estimated overlapping benefits and the actual operating state.
The scheduler collects hardware operating status data, dynamically evaluates the number of candidate overlap layers, and combines the handling delay correction amount and marginal benefit density to prioritize KV cache allocation and dynamically adjust the number of overlap layers to optimize resource utilization.
It effectively reduces DDR latency, reduces invalid on-chip reservations, lowers scheduling overhead, and prioritizes KV cache allocation under abnormal operating conditions, thereby improving resource utilization efficiency.
Smart Images

Figure CN123115045A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of heterogeneous storage and hardware resource scheduling technology in computer systems, and in particular to a storage scheduling method and system for the large model decoding stage. Background Technology
[0002] As large language models are increasingly deployed in online dialogue generation, long-context text processing, code completion, and inference services employing MoE or sliding window structures, the models typically need to perform forward computation layer by layer in hierarchical order during the decoding phase. In each generation round, they must read the weights required for the current layer and the key-value cache data related to the context length. In hardware systems employing in-memory computation (PIM) chips and DDR memory working in tandem, the PIM chip has high on-chip access bandwidth and can directly perform computations, but its on-chip memory capacity is relatively limited. DDR memory provides a large external storage space, but it requires transferring data from subsequent layers to the chip via DDR channels. To reduce PIM chip pauses due to waiting for data, the DDR controller can typically pre-transfer the weights or related data of subsequent layers while the PIM chip is computing the current layer, thus creating an overlap between computation and data transfer. However, common approaches in existing technologies involve pre-fixing the number of overlap layers during the compilation or deployment phase, or setting the prefetch depth solely based on the first round of decoding. In short-context scenarios, this approach may be overly conservative in its configuration, failing to fully utilize the remaining on-chip space. In long-context or multi-round decoding scenarios, the continuous growth of the KV cache may exacerbate on-chip space constraints. Furthermore, the residual data transfer between rounds during continuous decoding can alter the computation window that could have been used to hide DDR data transfers. If the initial window is still used for judgment, the estimated overlap benefit may not match the actual operating state. Therefore, there is an urgent need for a technical solution that can dynamically determine the number of overlap layers and manage the KV cache and weight prefetching in conjunction with the current context length, decoding rounds, remaining on-chip space, KV cache usage, DDR data transfer time, and PIM computation time. Summary of the Invention
[0003] The purpose of this application is to provide a storage scheduling method and system for the decoding stage of large models, so as to solve the problems mentioned in the background art.
[0004] This application provides a storage scheduling method for the decoding stage of large models. The method is executed by a scheduler and includes the following steps: Status acquisition steps: Acquire hardware operating status data, which includes the current context length, current decoding round, remaining on-chip allocable space of the PIM chip, weight of the layer to be calculated, KV cache size, measured handling time of DDR memory, and calculation time of the PIM chip. Candidate space evaluation steps: Based on the remaining on-chip allocable space in the hardware operating status data, the KV cache size, and the weight size corresponding to the candidate overlapping layers, the space feasibility of multiple preset candidate overlapping layers is judged, and the space feasible candidate overlapping layers are selected. Overlap benefit evaluation steps: Based on the remaining DDR transport time at the end of the previous decoding round and the idle time between rounds, update the transport delay correction amount for the current decoding round; based on the transport delay correction amount, correct the calculation window in the current decoding round that can be used to hide the DDR memory transport time; combined with the corrected calculation window, calculate the effective overlap benefit corresponding to each of the spatially feasible candidate overlap layers. Arbitration and selection steps: Based on the effective overlap benefits of each of the spatially feasible candidate overlap layers, calculate the corresponding marginal benefit density and select the target overlap layer from the spatially feasible candidate overlap layers; Scheduling control steps: Based on the target number of overlapping layers, control the memory management unit of the PIM chip to allocate or reserve KV cache pages, release or write back the weight space of low-yield overlapping layers, and send the corresponding weight prefetch instruction to the DDR controller.
[0005] In a preferred embodiment, during the overlapping benefit evaluation step: Regarding the current first Each decoding round, the transport delay correction amount Iterative updates are performed according to the following formula: During the first round of decoding ; This is the amount of delay correction for the previous decoding round; The remaining unfinished time for the last layer of the previous decoding round; The calculation from the end of the previous decoding round to the current round... The measured idle time of the DDR channel between the start of each decoding round; The function is used to maximize the value, ensuring that the transport delay correction is not negative; this applies when there is no gap in continuous decoding. In this case, the transport delay correction amount is used to characterize the cumulative encroachment of DDR transport residues on the prefetch window of subsequent rounds in consecutive decoding rounds; For the number of feasible candidate overlapping layers in the space Its corresponding effective overlapping benefits Calculate using the following formula: in, The layer number currently being calculated; The layer offset of the layer to be calculated; For the transport of the DDR memory Time required for layer weights; Calculate the first for the PIM chip Time required for each layer; The function is for finding the minimum value; For summation functions; The corrected calculation window represents the actual calculation time available for parallel processing after the processing delay correction. The effective overlapping benefits Indicates adoption The reduction in total DDR latency due to layer overlap compared to not using overlap.
[0006] In a preferred embodiment, during the arbitration and selection step: marginal revenue density Calculate using the following formula: in, The number of candidate overlapping layers is Effective overlapping benefits at the same time; For the first The weight of the layer; the marginal benefit density represents the additional transport hiding time gained for each additional unit of on-chip space occupancy; When the marginal revenue density When the marginal revenue density is less than the set lower threshold, the corresponding number of spatially feasible candidate overlapping layers is eliminated. The step of selecting the target overlap layer from the spatially feasible candidate overlap layers includes: selecting from the candidates that have not been eliminated, such that the effective overlap gain is maximized. The largest candidate is used as the target overlap layer number for the current decoding round. When the difference between the effective overlap benefit of the candidate selected in the current decoding round that maximizes the effective overlap benefit and the effective overlap benefit of the target overlap layer number in the previous decoding round is less than a set hysteresis threshold, the target overlap layer number of the previous decoding round is kept as the target overlap layer number for the current decoding round. This reduces frequent switching between adjacent decoding rounds caused by small fluctuations in benefits and the resulting prefetch cancellation and re-initiation overhead.
[0007] In a preferred embodiment, the arbitration and selection step further includes: under the constraint that the KV cache allocation priority is higher than the overlap layer weight retention priority, eliminating candidate overlap layers with insufficient returns based on the marginal return density, and selecting the target overlap layer from the candidate overlap layers that have not been eliminated; wherein, ensuring that the KV cache allocation priority is higher than the overlap layer weight retention priority includes executing the following spatial arbitration rules: When the remaining allocable space on the chip is less than the space required for the newly added KV cache, the overlapping layer weights are released in order of increasing marginal revenue density until the remaining allocable space on the chip meets the requirements of the newly added KV cache, so as to reduce the risk of KV cache allocation failure. The KV cache is allocated on a page-by-page basis, and when the current page is full, a new page is requested from the memory management unit of the PIM chip; When all overlapping layer weights are released, and the remaining on-chip allocable space is still less than the space required for the newly added KV cache, the least recently used KV cache cold pages are written back to the DDR memory and reloaded from the DDR memory when used in subsequent applications. The KV cache pages are divided into high-priority pages and low-priority pages based on the most recently accessed layer and the current context window position. When it is necessary to write back the KV cache pages, the low-priority pages are written back to the DDR memory first.
[0008] In a preferred embodiment, during the candidate space evaluation step: For any candidate number of overlapping layers Its corresponding space requirements Calculate using the following formula: in, This represents the total KV cache currently occupied on the PIM chip. For those that need to be prefetched The sum of the spatial weights of the layers. Calculate the required temporary buffer space for the current layer. The on-chip protection level is used to cover the newly added KV write volume and the space required for prefetch rollback. When the space requirement The candidate overlap layer number is determined when the on-chip memory capacity of the PIM chip is greater than the total on-chip memory capacity, or when the on-chip protection level is less than the amount of newly added KV data in this round. Spatial constraints are not met.
[0009] In a preferred embodiment, the method further includes a steady-state detection step: during continuous decoding, when the rate of change of the effective overlap gain corresponding to the target overlap layer number satisfies the steady-state convergence condition, it is determined to enter the decoding steady state, the target overlap layer number is frozen to stop the full evaluation in each round, and the space of the overlap layer weights with marginal gain density below a preset lower limit is released; when the re-evaluation condition is met, the decoding steady state is exited and the full evaluation is resumed.
[0010] In a preferred embodiment, during the steady-state detection step: The steady-state convergence condition for determining whether to enter the decoding steady state is that, in continuous... Within each decoding round, the target overlap layer number The rate of change of the corresponding effective overlap payoffs satisfies the following formula: in, The length of the steady-state observation window. To prevent positive numbers with a denominator of zero, The steady-state convergence threshold, The threshold parameter is determined by the specific hardware platform calibration and is used to calculate the absolute value function. The conditions for reassessment include any one of the following: The change in the current context length relative to the context length when entering the decoding steady state exceeds the set length change threshold; The number of pages retained in the on-chip key-value cache has increased beyond the set page increase threshold; The measured transport time of the DDR memory deviated from the predicted value by more than the set deviation threshold.
[0011] In a preferred embodiment, the method further includes a degradation handling step to address abnormal operating conditions: When the measured bandwidth of the DDR memory is lower than the set nominal value ratio threshold, the current target overlap layer number is halved and the evaluation cycle is shortened until the measured bandwidth is restored. If the prefetching of a certain layer's weights is not completed before the start of the layer's calculation and this happens consecutively a preset number of times, then the layer is marked as a prefetching failure layer, and no more than one layer's prefetch depth will be allocated to that layer in subsequent decoding rounds. When the memory management unit of the PIM chip fails to allocate new pages for the KV cache for a preset number of consecutive times, an emergency write-back is triggered. All KV cold pages with the lowest preset proportion of overlap layer weights and marginal benefit densities are simultaneously written back to the DDR memory, and the target overlap layer number is forcibly set to zero to enter the conservative retention mode.
[0012] This application provides a storage scheduling system for the large model decoding stage, including a scheduler, an in-memory computing PIM chip, a double data rate DDR memory, a DDR controller, and an on-chip memory management unit for the PIM chip. The scheduler includes the following cooperative functional modules: The status acquisition module is configured to acquire hardware operating status data, which includes the current context length, the current decoding round, the remaining on-chip allocable space of the PIM chip, the weight of the layer to be calculated, the KV cache size, the measured handling time of the DDR memory, and the calculation time of the PIM chip. The candidate space evaluation module is configured to judge the space feasibility of multiple preset candidate overlapping layers based on the remaining on-chip allocable space, the KV cache size, and the weight size corresponding to the candidate overlapping layers in the hardware operating status data, and to filter out the space-feasible candidate overlapping layers. The overlap benefit evaluation module is configured to update the transport delay correction amount of the current decoding round based on the residual time of DDR transport that has not yet been completed at the end of the previous decoding round and the idle time between rounds; based on the transport delay correction amount, correct the calculation window in the current decoding round that can be used to hide the DDR memory transport time; and combine the corrected calculation window to calculate the effective overlap benefit corresponding to each of the spatially feasible candidate overlap layers. The arbitration and selection module is configured to calculate the corresponding marginal benefit density and select the target number of overlapping layers from the spatially feasible candidate overlapping layers based on the effective overlapping benefits of each of the spatially feasible candidate overlapping layers. The scheduling control module is configured to control the on-chip memory management unit to perform the allocation or retention of KV cache pages, the release or write-back of the weight space of low-yield overlapping layers, and to issue a weight prefetch instruction for the corresponding layer to be calculated to the DDR controller, based on the target number of overlapping layers.
[0013] In a preferred embodiment, the scheduler further includes a steady-state detection module, configured to, during continuous decoding, determine to enter the decoding steady state when the rate of change of the effective overlap gain corresponding to the target overlap layer number satisfies the steady-state convergence condition, freeze the target overlap layer number to stop the full evaluation in each round, and release the space of the overlap layer weights whose marginal gain density is lower than a preset lower limit; when the re-evaluation condition is met, exit the decoding steady state and resume the full evaluation.
[0014] This application provides an electronic device, including: At least one processor; The in-memory computing PIM chip has on-chip memory space; Double Data Rate (DDR) memory; and a memory communicatively connected to the at least one processor; The memory stores computer program instructions, which, when executed by the at least one processor, implement the aforementioned memory scheduling method for the large model decoding stage.
[0015] This application provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the steps of the above-described storage scheduling method for the large model decoding stage.
[0016] By assessing the spatial feasibility of the candidate number of overlapping layers, and using... Characterized by total KV cache usage, prefetch weight space, temporary buffer space, and on-chip protection water level. Based on the overall space requirements, the scheduler can eliminate candidate schemes that may encroach on the space for new KV writes or prefetch rollback before selecting the target number of overlap layers. This approach ensures that overlap layer retention is no longer solely determined by a localized decision of "how many layers can be moved in advance," but is pre-selected under the constraints of the total on-chip capacity of the PIM and the currently allocable space. This helps reduce the risk of KV cache allocation or subsequent computation buffer usage being affected by prefetch weights occupying too much on-chip space.
[0017] By introducing a handling delay correction And based on the remaining DDR transport time that was not completed at the end of the previous decoding round. and inter-wheel idle time The hidden transport window for the current decoding round is modified, and the proposed technical solution can reflect the temporal coupling relationship between transport tasks in successive decoding rounds. Because... It does not decrease monotonically during continuous, gapless decoding, and has effective overlap gains. The calculation can reflect the physical trend of the overlap window being gradually compressed from its initial size during the first round of decoding. This reduces the risk of overestimating benefits caused by directly applying the first round of decoding or the ideal parallel window to subsequent rounds, making the determination of the subsequent target overlap layer number more consistent with the degree of overlap between PIM calculation and DDR transport in the actual timing. If lacking... If the scheduler continues to estimate the benefit based on the ideal window of the first round of decoding, it may continue to retain the overlapping layer weights that can no longer effectively hide DDR transport after multiple rounds of decoding, resulting in the PIM on-chip space being ineffectively occupied.
[0018] By setting the marginal revenue density The scheduler can transform the decision of "whether to add or retain an additional layer of overlap weight" into an evaluation of the marginal hiding time per unit of on-chip space. Therefore, when on-chip space is limited, the selection is not simply based on the number of layers or the remaining capacity, but rather prioritizes identifying and eliminating low-yield-density overlap layers, ensuring that limited on-chip space is allocated more towards data that is more valuable for hiding DDR latency. In short-context scenarios with ample on-chip space, many candidates can be retained based on both space feasibility and marginal yield density, potentially improving decoding speed by utilizing surplus space. However, in long-context scenarios where KV cache growth leads to space constraints, candidates with lower marginal yield density will be prioritized for elimination, allowing space to be allocated to data with higher yield. This is combined with a hysteresis threshold. Furthermore, maintaining the target overlap layer of the previous round when the difference in benefits between adjacent decoding rounds is small also helps to reduce the additional scheduling overhead caused by prefetch cancellation, re-initiation, and frequent switching. As a compromise, the cost is that it may briefly stay on the suboptimal candidate near the switching boundary, but the hysteresis rule no longer applies when the KV cache allocation is insufficient or abnormal degradation conditions are triggered, so as to prioritize satisfying the rigid space constraints of inference execution.
[0019] By setting the KV cache allocation priority higher than the overlap layer weight retention priority, and releasing overlap layer weights from low to high based on marginal benefit density when on-chip remaining allocable space is insufficient, this technical solution incorporates the rigid requirement of continuous KV cache growth into the overlap layer selection process. KV cache is allocated page-by-page, with low-priority pages or cold pages written back to DDR when necessary and reloaded upon subsequent access. This provides a fallback mechanism for the scheduler under extreme space pressure, reducing the risk of KV cache allocation failure due to excessive retention of overlap layer weights. Furthermore, KV cache pages are divided into high-priority and low-priority pages based on the most recently accessed layer and the current context window position. Low-priority pages are evicted first when write-back is required, helping to reduce the additional handling overhead caused by frequent readbacks during subsequent attention accesses. This mechanism reflects a constrained optimization of overlap benefits while ensuring that inference-necessary cache space receives on-chip resources first, rather than optimizing KV cache usage and overlap benefits independently.
[0020] After determining the target number of overlap layers, the scheduling control module controls the on-chip memory management unit of the PIM chip to allocate or reserve KV cache pages, release or write back the weight space of low-yield overlap layers, and issue weight prefetch instructions for the corresponding layers to be calculated to the DDR controller. , and This can be translated into concrete data movement and on-chip space management actions. In this way, a clear causal relationship is established between formula calculation and hardware resource scheduling, which helps to demonstrate the improvement of the solution on the internal data storage, data movement and hardware execution processes of the computer system, rather than remaining at the level of abstract evaluation indicators.
[0021] By monitoring the rate of change of effective overlap gain corresponding to the target overlap layer number during continuous decoding, and freezing the target overlap layer number, stopping each round of full evaluation, and releasing the weight space of overlap layers with marginal gain density below a preset lower limit when the steady-state convergence condition is met, this technical solution can reduce the scheduling overhead caused by repeated evaluation after the gain tends to stabilize, and allow the on-chip space occupied by low-gain layers to be reused for KV cache or other necessary data. When the context length, KV cache page count, or DDR measured transfer time changes in a way that is sufficient to affect the original judgment, exiting the decoding steady state and resuming full evaluation helps to avoid continuing to use an inappropriate overlap layer number after a sudden change in the running state due to steady-state freezing.
[0022] By setting degradation handling logic under abnormal operating conditions, the scheduler can reduce the target overlap layer number, limit the depth of prefetching of failed layers, trigger overlap layer weights and write back some KV cold pages when the measured DDR bandwidth decreases, specific layer prefetching fails repeatedly, or the PIM on-chip memory management unit fails to allocate new pages for the KV cache multiple times. It can also enter a conservative retention mode when necessary. This approach helps reduce the repeated initiation of low-success-rate prefetching or the continuous occupation of low-yield on-chip space under abnormal conditions, allowing the system to prioritize maintaining necessary KV cache and computational data requirements even under non-ideal operating conditions. By supporting lookup table evaluation mode and segmented evaluation mode, the technical solution can also reduce the number of online full calculations in scenarios with limited hardware resources or many decoding rounds: the lookup table method can quickly obtain the target overlap layer number using the current context length, decoding rounds, and remaining on-chip allocable space, and revert to full calculation when the measured deviation is large; the segmented method can use different evaluation frequencies in the startup, transition, and steady-state stages, helping to achieve an adjustable trade-off between scheduling accuracy and scheduling overhead.
[0023] The corresponding system, through the cooperation of the state acquisition module, candidate space evaluation module, overlap benefit evaluation module, arbitration and selection module, scheduling control module, and steady-state detection module, distributes the state perception, candidate screening, window correction, benefit evaluation, spatial arbitration, and hardware action distribution from the above methods to implementable functional modules. The configuration of electronic devices and computer-readable storage media enables this scheduling logic to be deployed as program instructions in a hardware system including PIM chips, DDR memory, and processors, thus providing a foundation for heterogeneous storage scheduling in the large model decoding stage under different deployment configurations.
[0024] In summary, the above-mentioned links form a closed-loop feedback system, with state data, candidate overlap layer number, effective overlap benefit, marginal benefit density, and scheduling control instructions as the transmission objects: transport delay correction amount. The cumulative effect of each round provides a benefit input reflecting the actual window change for the calculation of marginal benefit density. The elimination result of marginal benefit density prioritizes the allocation of KV cache pages to free up on-chip space. Steady-state determination controls the execution frequency of the above full evaluation, while abnormal degradation and multiple exit conditions break the freeze and reactivate the closed loop when the operating conditions change abruptly. The overall technical effect of this closed-loop scheduling process—that is, under the multiple constraints of limited on-chip memory capacity of PIM, dynamic growth of KV cache with context and gradual accumulation of transport residue between consecutive decoding rounds, it helps to reduce DDR latency, reduce invalid on-chip retention and reduce scheduling overhead under the rigid constraint of ensuring KV cache allocability—is not a simple sum of the effects of individual technical means, but a synergistic result that depends on the specific causal cascade relationship and mutual constraint mode between the above transport delay correction, marginal benefit elimination, KV priority arbitration and steady-state convergence determination.
[0025] The specification of this application contains numerous technical features distributed across various technical solutions. Listing all possible combinations of these technical features (i.e., technical solutions) would make the specification excessively lengthy. To avoid this problem, the various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which should be considered as described in this specification), unless such a combination of technical features is technically infeasible. For example, one example discloses feature A+B+C, and another example discloses feature A+B+D+E. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; they cannot be used simultaneously. Feature E can technically be combined with feature C. Therefore, the solution A+B+C+D should not be considered as described because it is technically infeasible, while the solution A+B+C+E should be considered as described. Attached Figure Description
[0026] Figure 1 This is a flowchart illustrating the storage scheduling method for the large model decoding stage provided in this application embodiment, showing the processing flow between state acquisition, candidate space evaluation, overlap benefit evaluation, arbitration and selection, scheduling control, steady-state detection, and abnormal condition degradation processing.
[0027] Figure 2This is a schematic diagram of the storage scheduling system for the large model decoding stage provided in this application embodiment. It shows the connection relationship between the scheduler, the in-memory computing PIM chip, the on-chip memory management unit, the double data rate DDR memory, the DDR controller, and the state acquisition module, the candidate space evaluation module, the overlap gain evaluation module, the arbitration and selection module, the scheduling control module, and the steady-state detection module. Detailed Implementation
[0028] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions of this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.
[0029] Explanation of some concepts: The large model decoding stage refers to the reasoning stage in which the large language model generates output tokens step by step according to the decoding rounds after completing the pre-processing of the input, and performs forward computation layer by layer in each decoding round.
[0030] A PIM chip refers to a Processing-in-Memory (PIM) chip that has high-bandwidth on-chip memory and can perform computations on-chip or near the storage side. For example, it can be a chip using a three-dimensional stacking process, which has the physical characteristics of relatively small on-chip memory capacity but high access bandwidth.
[0031] DDR memory refers to an external double data rate memory used to store model weights, KV cache, or other inference data, and to transfer data to the PIM chip through the DDR channel.
[0032] The number of overlapping layers refers to the number of data layers to be calculated that are additionally reserved or pre-fetched on the PIM chip in order to enable PIM calculation and DDR transport to be executed in parallel. In this application, "number of overlapping layers", "candidate number of overlapping layers" and "target number of overlapping layers" have the same meaning.
[0033] Candidate overlap layers refer to one or more alternative overlap layers that the scheduler uses for spatial feasibility assessment and benefit assessment during the current assessment period.
[0034] Number of target overlap layers This refers to the actual number of overlap layers used by the scheduler in the current decoding round, based on spatial constraints, effective overlap benefits, marginal benefit density, and KV cache priority constraints.
[0035] The effective overlap window refers to the computation time window that can actually be used to hide DDR transport time during the PIM chip's execution of calculations in the current or adjacent layers.
[0036] Transportation delay correction , refers to the time correction amount used to characterize the degree of encroachment of the prefetch window of the current decoding round by the DDR transport task that has not yet been completed at the end of the previous decoding round during continuous decoding. It is updated iteratively with each decoding round.
[0037] Effective overlapping benefits , refers to the first Each decoding round uses the candidate overlap layer number At that time, the DDR latency reduction can be achieved by not using overlapping transport.
[0038] Marginal revenue density This refers to the additional transport and hiding time gained for each additional unit of on-chip space occupied, used to determine whether adding or retaining the corresponding overlapping layer has sufficient on-chip space utilization value.
[0039] KV cache refers to cached data saved during the decoding stage of a large model to reuse key vectors and value vectors from the historical context. Its size increases linearly with the length of the context.
[0040] A key-value cache page refers to a data unit that is allocated, retained, released, or written back to on-chip according to the page, block, or other granularity of the key-value cache.
[0041] Remaining allocatable space on the chip This refers to the space in the PIM chip's on-chip memory that can currently be allocated by the scheduler to the KV cache, weight prefetch, or temporary buffer.
[0042] Space requirements This refers to using the number of candidate overlap layers. At that time, the KV cache, the weight to be prefetched, the temporary buffer, and the on-chip protection level together require the on-chip space.
[0043] Protected water level on film This refers to the on-chip space reserved for new KV writing and prefetch rollback operations in this round.
[0044] Decoding steady state refers to the operational state in which the change in effective overlap revenue tends to stabilize during continuous decoding, and the marginal value of continuing full evaluation or continuing to retain low-return overlap layers decreases.
[0045] Hysteresis threshold , refers to the threshold used to suppress frequent switching of the target overlap layer between adjacent decoding rounds due to small fluctuations in effective overlap gains.
[0046] Conservative retention mode refers to the operating mode that reduces or suspends the retention of multiple layers of overlap when the benefits of candidate overlap are insufficient, space is not feasible, or abnormal operating conditions occur, and prioritizes the allocation of on-chip space to KV cache pages and necessary computation buffers.
[0047] The following is a brief summary of some of the innovative aspects of this application: In summary, in the PIM and DDR heterogeneous memory scheduling scenario for the large model decoding stage, this application does not merely change the number of overlap layers from a fixed value to a runtime variable, but rather addresses the mismatch problem of static configuration in short contexts, long contexts, and multi-round decoding, and addresses it in a way that... Figure 1 The illustrated state acquisition and layer selection process, the convergence of the effective overlap window to a steady state, and the relationship between KV cache growth and on-chip space contention are organized into a closed-loop scheduling concept governed by the same on-chip resource constraints. The scheduler obtains the current context length. Decoding rounds Remaining allocatable space on the chip KV cache usage Weight size DDR transport time PIM calculation time Next, let's consider the space requirements. and protection water level Eliminating candidates that would encroach on the newly added KV writing space, and then considering the residual material transported at the rear of the front wheel... Inter-wheel idle time Iterative update of transport delay correction amount (Formula 4) Correct the ideal window to --This recursive correction method for quantifying "return decay with each round" is not found in existing fixed overlap or ping-pong buffer schemes. Its physical meaning lies in characterizing the temporal encroachment of the previous round's tail DDR transport residue on the current round's prefetch window. Furthermore, marginal return density... (Formula 6) transforms "whether to retain an additional layer of weight" into an evaluation of the marginal hidden time corresponding to each byte of on-chip space, thereby increasing the target overlap layer number. Effective overlapping benefits (Formula 5), a constrained choice is formed between KV cache priority arbitration (step 450) and spatial feasibility; hysteresis threshold The steady-state convergence determination (Formula 8, Step 600) further suppresses repeated switching during small fluctuations in returns and ineffective full evaluation after steady state. Crucially, these mechanisms do not operate independently but form a causal cascade: The cumulative effect of each round Provides revised revenue inputs. The elimination process prioritizes the allocation of KV cache pages to free up on-chip space. Steady-state determination controls the frequency of full evaluation and reactivates it when the context or DDR transport state changes abruptly. Thus, weight prefetching, KV page retention or write-back, release of low-yield layers, and steady-state freezing form a chain of hardware actions that are interconnected. This is not a simple application of "adaptive scheduling based on load status," but rather an organic integration that addresses the three technical contradictions of limited on-chip capacity of PIM, dynamic growth of KV cache with context, and gradual accumulation of transport residue between consecutive decoding rounds.
[0048] Furthermore, analysis revealed that the performance bottleneck in the large-model decoding stage of the PIM and DDR heterogeneous memory system does not simply stem from insufficient DDR channel bandwidth or small PIM on-chip capacity. Rather, it arises from the dynamic relationship of continuously changing and interdependent factors among "transfer time window, on-chip space occupancy, and KV cache growth" during continuous decoding. Further analysis shows that the instability of a fixed number of overlap layers under different context lengths originates from the fact that its default on-chip reservable space and hidden transfer window remain relatively constant during operation. However, during actual decoding, the KV cache gradually occupies on-chip space as the context grows. Unfinished DDR transfers at the end of the previous decoding round may also encroach on the prefetch window of the current round. This means that what appears to be a feasible multi-layer overlap in the first round of decoding may transform into low-yield or even ineffective on-chip reservations in subsequent rounds. Furthermore, if the prefetch depth is determined solely based on the remaining space, it is impossible to identify how much DDR wait time a candidate overlap layer can actually hide—the lack of a quantitative method to characterize the cumulative effect of cross-round transport remnants leads to a systematic overestimation of the overlap benefits in subsequent rounds. If the benefits are estimated solely based on DDR transport time or PIM computation time, the rigid constraint of KV cache page allocation may be ignored, causing weight retention and KV writing to compete within the same on-chip space and preventing release according to benefit priority. More importantly, the change in effective overlap benefits gradually stabilizes after continuous decoding. Performing a full evaluation in each round at this point incurs unnecessary computational overhead. However, simply fixing the overlap layer number permanently prevents timely recovery and adjustment when context length changes, KV cache page counts surge, or DDR channel states change. Therefore, this application abstracts the previous round transport remnant and inter-round idle time as transport delay correction amounts. To correct the computation window available for hiding transport in the current round; and through effective overlap of benefits. and marginal revenue density The process transforms the decision of "whether to add or release a certain overlap layer" into a comparable evaluation constrained by on-chip space. Under the constraint of priority allocation of KV cache, the target number of overlap layers is selected, and scheduling is frozen when the benefits tend to stabilize, and evaluation is resumed when the context or transport state changes. The implementation process of this application is described in detail below through specific embodiments.
[0049] To make the purpose, technical solution, and advantages of this application clearer, the following is in conjunction with... Figure 1 and Figure 2 The embodiments of this application are clearly and completely described. It should be understood that the following embodiments are only used to illustrate the technical concept of this application and should not be construed as limiting the scope of protection of this application. Without departing from the technical concept of this application, those skilled in the art can make adaptive adjustments to the relevant thresholds, candidate sets, and evaluation cycles according to the specific PIM chip structure, DDR channel bandwidth, number of model layers, KV cache organization method, and deployment environment, and other implementation methods obtained are all within the scope of protection of this application. It should also be noted that, for ease of description, the term "number of overlapping layers" is used below to uniformly refer to the number of subsequent layers to be calculated that are additionally reserved or pre-fetched on the PIM chip to enable parallel computation and transport. In different paragraphs, it has the same meaning as "candidate number of overlapping layers" and "target number of overlapping layers" and will not be repeated separately.
[0050] I. Application Environment and System Architecture The method provided in this application is applied to an inference system comprising a scheduler, a Processing-in-Memory (PIM) chip, a Double Data Rate (DDR) memory, a DDR controller, and an on-chip memory management unit of the PIM chip. For example... Figure 2 As shown, the PIM chip is an in-memory computing chip with on-chip high-bandwidth memory and the ability to directly perform some inference calculations on-chip. For example, it can be a chip using a three-dimensional stacking process, possessing the physical characteristics of relatively small on-chip memory capacity but high access bandwidth. In this embodiment, its total on-chip memory capacity is denoted as... The DDR memory, as a large-capacity external memory, is used to store model weights, KV cache, or other intermediate data, and transfers data with the PIM chip via the DDR channel under the management of the DDR controller.
[0051] During the decoding phase of a large language model, the model typically generates output word by word. Each round of decoding requires performing forward computation layer by layer according to the model's layer order. The computation of each layer requires loading the weights of that layer and the corresponding key-value cache data. Due to the limited on-chip memory capacity, not all weights and key-value caches can reside on the chip. Therefore, while the PIM chip is computing the current layer, the DDR controller can pre-load data for the subsequent layers to be computed. This allows for overlapping execution of computation and data loading, minimizing the possibility of the PIM chip pausing due to waiting for data.
[0052] Unlike methods that fix the number of overlap layers all at once during the compilation or deployment phase, this embodiment dynamically determines the target number of overlap layers for the current decoding round at runtime based on the current context length, the current decoding round, the remaining on-chip allocable space of the PIM chip, the weight of the layer to be computed, the KV cache size, the measured DDR memory transfer time, and the PIM chip's computation time. It should be noted that since the KV cache continuously occupies on-chip space as the context length increases during continuous decoding, and there may be tail remnants in the DDR transfer tasks between decoding rounds, this embodiment further integrates the allocation of KV cache pages, the retention of overlap layer weights, the release of low-yield overlap layer weights, and steady-state detection into a single scheduling loop.
[0053] Specifically, the method can be executed by a scheduler. It should be noted that the scheduler can be implemented as a software scheduler running on the processor, as hardware control logic in a PIM chip or inference acceleration system, or as a software-hardware co-operating scheduling unit. The scheduler's output is not a simple abstract score, but a control result that drives hardware execution, including the target overlap layer number, KV cache page allocation or retention instructions, overlap layer weight release or write-back instructions, and weight prefetch instructions for subsequent layers to be computed. The core of this embodiment is that the overlap layer number is not a fixed static value, but is dynamically determined by the scheduler during runtime based on the aforementioned hardware operating status data, and is processed in conjunction with the dynamic allocation of the KV cache, thereby achieving a more suitable on-chip space utilization under different context lengths and different decoding stages.
[0054] II. Method Examples This embodiment provides a storage scheduling method for the large model decoding stage, executed by the scheduler. For example... Figure 1 As shown, the main steps include state acquisition (step 100), candidate space evaluation (step 200), overlap benefit evaluation (step 300), arbitration and selection (step 400), scheduling control (step 500), and steady-state detection (step 600), and further include degradation processing under abnormal operating conditions (step 700). Each step will be explained in detail below.
[0055] Step 100: Runtime Hardware Status Acquisition At the start of each evaluation cycle, the scheduler collects current hardware operating status data. Specifically, the collected parameters include: current context length. This can represent the number of lexical units that have participated in the attention calculation for the current request; the current decoding round. This indicates the number of decoding rounds counted from the end of the prefill phase. In this embodiment, the first decoding round after the prefill phase ends is counted as [number]. The remaining allocatable space on the PIM chip The weights of the layers to be computed can be maintained by the on-chip memory management unit. This can come from the model layer configuration table; Layer KV cache size DDR memory transport Measured transport time required for layer weights This can come from transport statistics or hardware counters from the DDR controller; and from the PIM chip calculating the first... Computation time required for each layer The results can come from static performance profiling, historical averages, or current hardware timings.
[0056] Furthermore, the first Layer KV cache size It can be determined by the current context length and the number of attention heads. Attention head dimension and data precision in bytes Together, they can be determined, for example, by estimating using the following formula: In this formula, coefficient 2 represents both the key cache and the value cache. The length of the current context as mentioned above; The dimension of a single attention head; The number of heads to focus on; This represents the number of bytes occupied by a single element at the current data precision. As shown in Equation 1, the KV cache size increases linearly with the context length, which is the physical reason why on-chip space tends to be tight in long context scenarios. It should be noted that Equation 1 is used to illustrate the correspondence between the KV cache size and the context length, and can be adjusted according to the actual KV cache organization method under different model structures or different KV cache compression strategies; optionally, the scheduler can also directly read the actual KV cache usage obtained from the on-chip memory management unit statistics, without having to recalculate it using the formula in each round.
[0057] Furthermore, DDR transport time Can be derived from the first Layer weight size and DDR bandwidth It is determined that, for example, it can be estimated using the following formula: in, For the aforementioned Layer weight size; This refers to the measured or estimated bandwidth within the current evaluation period. It should be noted that the method for obtaining the above time parameters is not unique. It can be derived from bandwidth-based estimation (such as Equation 2), or it can be obtained by the DDR controller based on the measured time consumption of the most recent transport tasks. When using the measured value, Equation 2 can be used as a predicted value or a verification value to determine whether the DDR channel status has changed significantly.
[0058] Optionally, to improve the robustness of the data acquisition process, when any of the above parameters fails to be acquired (e.g., hardware counter read timeout), the scheduler marks the layer as "data unavailable" and replaces it with the historical average of the layer in the most recent rounds in subsequent evaluations, while triggering a hardware status self-check, thereby avoiding the interruption of the entire evaluation cycle due to a single point of acquisition failure.
[0059] Step 200: Spatial Feasibility Assessment of Candidate Overlapping Layers The scheduler performs a spatial feasibility assessment on multiple candidate overlap layers, either preset or dynamically generated. Specifically, the candidate overlap layers can be a preset set or dynamically generated based on the current remaining space and model layer order. For example, the candidate set may include several levels such as small overlap layers, medium overlap layers, and large overlap layers, to select different on-chip retention strategies under different operating states such as short context, long context, and continuous decoding steady state. Optionally, the candidate set can be given in increments of larger steps (e.g., powers of 2), which reduces the number of candidate evaluations, thereby reducing scheduling overhead, and facilitates matching with hardware transport alignment boundaries. Those skilled in the art will understand that the specific values of the candidate set can be adaptively adjusted according to the number of model layers and hardware configuration, and are not limited to the above examples.
[0060] Furthermore, for any candidate number of overlapping layers The scheduler calculates its corresponding on-chip space requirement, which can be estimated by the following formula: in, This represents the total KV cache currently stored on the PIM chip. It should be noted that this differs from the single-layer KV cache in Formula 1. The meanings are different; the former refers to the sum of all key-value caches stored on the chip. For those that need to be pre-fetched or reserved The sum of the spatial weights of each layer to be computed. This is the layer number currently being calculated. The offset of the layer to be calculated relative to the current layer; Calculate the temporary buffer space required for the current layer; This is the on-chip protection level, designed to at least cover the KV write volume of newly added terms in this round and the space required for one prefetch rollback. Specifically, it sets... Its purpose is to reserve space for the newly added KV data in this round and retain the margin for rollback in case of a prefetch failure, so as to avoid the on-chip space being filled with the weight of the overlapping layer and there is nowhere to write the new KV.
[0061] When satisfied Or the on-chip protection water level Less than the amount of new KV data in this round At that time, the scheduler determines the number of candidate overlap layers. If a candidate does not meet the spatial constraints, it is removed from the candidate set and moved to a smaller set of candidates; otherwise, it is retained as a spatially feasible candidate overlapping layer. Through this step, subsequent benefit evaluation is performed only on spatially feasible candidate overlapping layers, thereby reducing the overhead of ineffective evaluation.
[0062] Step 300: Assessment of Transportation Delay Correction and Effective Overlap Benefits One of the key innovations of this embodiment is that, before evaluating the benefits of each candidate number of overlap layers, a transport delay correction is applied to the "computation window available for hiding DDR transport" to reflect the physical trend of the overlap window being gradually compressed during continuous decoding. Specifically, during continuous decoding, DDR transport tasks that have not yet been completed at the end of the previous decoding round may encroach on the time window available for prefetching at the beginning of the current decoding round; if the overlap benefit is still estimated based on the ideal state of the first round of decoding, the time available for hiding DDR transport in subsequent rounds is easily overestimated. Therefore, this step is further divided into the following sub-steps.
[0063] Step 310, update the handling delay correction amount. The handling delay correction amount... This characterizes the time encroachment of DDR transport remnants between consecutive decoding rounds on the prefetch window of the current round. In the first decoding round... At that time, take This indicates that there is no residual data from previous decoding rounds; during continuous decoding, Iterative updates are performed using the following formula: in, This is the amount of delay correction for the previous decoding round; The remaining unfinished time for the last layer of the previous decoding round; From the end of the previous decoding round to the current round The measured idle time of the DDR channel between the start of each decoding round; A function for maximizing the value is used to ensure that the transport delay correction is not negative. More specifically, The physical meaning is: the DDR transport task that was not completed at the end of the previous decoding round occupies the transport time window at the beginning of the current round, resulting in a corresponding reduction in the actual computation time available for hidden transport in the current round. For example, when decoding is continuous without gaps, i.e. At that time, it can be seen from Formula 4 that The monotonous process continues unabated and gradually accumulates with each round; when the residual time of each layer's handling and the calculation time tend to stabilize, It may tend towards a relatively stable value; this cumulative process characterizes the trend of a larger overlapping window during the first round of decoding, with the window gradually being moved and compressed after continuous decoding; conversely, if there is sufficient idle time between rounds, the residual effect can be partially or completely offset. It should be noted that when When the accumulated impact causes the corrected computation window to decrease to zero, the effective overlap gain for the corresponding number of candidate overlap layers will also decrease to zero. The scheduler will then either eliminate candidates based on marginal gain density or enter a conservative retention mode for degradation. The correction does not promise to eliminate all DDR waits, but rather makes a conservative adjustment to the selection of the number of overlap layers. Therefore, the scheduler can adjust the computation window available for hiding DDR moves in the current round based on the actual hardware timing, rather than simply using the static overlap window.
[0064] Step 320: Calculate the effective overlap benefit. Number of feasible candidate overlap layers for each space. The scheduler calculates its effective overlap gain by combining the modified calculation window. Defined as using The reduction in total DDR latency due to layer overlap compared to no overlap (i.e., 0-layer overlap) can be calculated, for example, by the following formula: in, For transporting the first Time required for layer weights; To calculate the first Time required for each layer; The function is for finding the minimum value; For summation functions; and The meaning is the same as before. Specifically, the internal logic of Formula 5 is: the PIM chip calculates the first... Layers can be moved in parallel. Layer data, but correction amount due to transport delay The existence of this means that the computational window actually available for parallel processing is determined by the ideal... Reduced to This refers to the corrected calculation window; and the actual hidden transport time of this layer cannot exceed the DDR transport time of the layer's weight. The time consumed cannot exceed the modified calculation window, so the smaller of the two values is taken, and the values of each layer within the candidate overlap range are accumulated. Thus, Formula 5 considers both the upper limit of the time consumed by the transport itself and the actual window after the parallel computing time is occupied by the transport residue, thereby avoiding overestimation of the effective overlap window.
[0065] Step 400: Marginal Revenue Density Arbitration and Target Layer Selection When on-chip space is limited in PIM (Polymer Memory Injection), not every additional layer of weight is equally valuable. Some overlapping layers, while occupying more on-chip space, have limited DDR (DDR) latency for further hiding due to the reduced computation window after transport delay correction. Therefore, this embodiment transforms the originally qualitative decision of "whether to retain an additional layer of weight" into a quantitative comparison of the marginal hiding time gained per unit of on-chip space. This step further includes the following processing.
[0066] Step 410: Calculate the marginal revenue density. This applies to the number of spatially feasible candidate overlapping layers. The scheduler calculates its marginal revenue density, which can be exemplarily calculated using the following formula: in, Take the number of candidate overlapping layers Effective overlapping benefits at the same time; For the first The weight of the layer. More specifically, the molecular weight. The characterization is by the first The incremental hiding time caused by layer overlap, denominator This represents the additional on-chip space occupied to retain this layer, therefore Physically, it represents "the additional hidden time of transportation gained by each additional unit of on-chip space occupied," that is, a space-to-time conversion rate.
[0067] Step 420: Eliminate candidate layers with insufficient returns. When a certain number of candidate overlapping layers... Corresponding marginal revenue density When the marginal return density is less than the set lower limit threshold, the scheduler determines the first... If the benefit of overlapping layers is insufficient, the corresponding candidate overlapping layers will be eliminated or the number of overlapping layers will no longer be increased. It should be noted that the lower limit threshold of marginal benefit density can be determined by hardware platform calibration, operational statistics or deployment strategies. Its purpose is not to pursue global optimization in an abstract sense, but to provide a discriminatory basis for releasing the weight of low-benefit overlapping layers when space on the PIM chip is tight.
[0068] Step 430: Select the target overlap layer. The scheduler selects the candidate with the best effective overlap benefit from all candidate overlap layers that meet the spatial feasibility condition and whose marginal benefit density meets the preset condition, as the target overlap layer for the current decoding round. For example, it can be represented as: (Formula 7) in, The set of candidate overlapping layers that simultaneously satisfy spatial feasibility and have not been eliminated; This represents the independent variable that maximizes the objective function. It should be noted that Formula 7 is one specific method for selecting the target overlap layer number. In other embodiments, the scheduler can also select candidates that satisfy space constraints and whose effective overlap gains reach preset conditions, without strictly selecting the mathematically maximum value. Furthermore, to reduce frequent switching between adjacent decoding rounds due to small fluctuations in gains, the scheduler can introduce a hysteresis threshold: when the difference between the effective overlap gain corresponding to the candidate selected in the current round and the effective overlap gain of the target overlap layer number used in the previous decoding round is less than the set hysteresis threshold, the scheduler keeps the target overlap layer number of the previous decoding round unchanged. Specifically, the purpose of introducing the hysteresis threshold is to reduce the control overhead caused by prefetch cancellation, re-initiating prefetching, and repeated allocation and release of on-chip space due to frequent switching; as a trade-off, the cost is that it may briefly linger on a suboptimal candidate with similar gains near the switching boundary, which is explicitly stated in this embodiment. It should be particularly noted that when insufficient KV cache allocation, unmet space constraints, or abnormal degradation conditions are triggered, the hysteresis rule no longer applies, prioritizing the rigid space constraints required for inference execution.
[0069] Step 440, Conservative Retention Mode. It should be noted that when all candidates do not meet the marginal revenue density condition, or all candidates are spatially infeasible, or after spatial feasibility and marginal revenue density judgments, there are no effective candidate overlapping layers, the scheduler sets the target overlapping layer number to 0 and enters the conservative retention mode: only the weight of the layer to be calculated is retained, and on-chip space is preferentially allocated to KV cache pages and computation temporary buffers.
[0070] Step 450: Joint spatial arbitration of KV cache and overlap layer weights. When the remaining on-chip space is limited, there is a space competition relationship between the KV cache and the overlap layer weights. This embodiment treats KV cache allocation as a rigid constraint with higher priority than the retention of overlap layer weights, and optimizes the effective overlap benefits brought by the number of overlap layers under this constraint. Specifically, firstly, when the remaining allocable space on the chip... Less than the space required for the newly added KV cache At that time, the scheduler releases the overlapping layer weights sequentially in order of increasing marginal revenue density, and controls the on-chip memory management unit to reclaim the corresponding space, until... This reduces the risk of KV cache allocation failure due to excessive retention of overlapping layer weights; secondly, the KV cache is allocated on a page-by-page basis, and no new page is requested when the current page is not full, and a new page is requested from the on-chip memory management unit when the current page is full, and the remaining on-chip allocable space is updated accordingly; thirdly, after all overlapping layer weights are released... If the cache is still insufficient, the scheduler writes the least recently used KV cache pages back to DDR memory and marks them as cold pages or DDR-resident pages, then reloads them from DDR memory when needed. Optionally, the KV cache pages can be further divided into high-priority pages and low-priority pages based on the most recently accessed layer and the current context window position. Pages within the current attention window or with a high recent access frequency are reserved as high-priority pages on-chip, while low-priority pages outside the current attention window or with a low recent access frequency are evicted when a write-back is needed, to reduce the extra data movement caused by frequent readbacks during subsequent attention accesses. It should be noted that the above hierarchical management does not change the overall approach of joint processing of dynamic KV cache allocation and overlap layer selection, but rather provides a more granular fallback for extreme space-constrained scenarios.
[0071] It should be further noted that the above-mentioned process of releasing all overlap layer weights and rewriting back to KV cold pages, in extremely long context scenarios, will require subsequent decoding rounds to reload these cold pages from DDR, thereby increasing the DDR handling burden and potentially further increasing costs. This embodiment indicates that the deterioration path has a natural termination condition, namely, when the number of overlapping layers decreases to 0. The accumulation rate will slow down significantly, so the technical effect of this solution should be characterized as achieving a better compromise under spatial constraints, rather than eliminating spatial constraints.
[0072] Step 500: Scheduling control and hardware action issuance The scheduler determines the target overlap layer number. Based on the aforementioned arbitration results, the on-chip memory management unit of the PIM chip is controlled to perform corresponding hardware actions. Specifically, the scheduler determines the weights of subsequent layers to be calculated that need to be prefetched or retained based on the current layer number and the target overlap layer number, and issues corresponding weight prefetch instructions to the DDR controller; for overlap layer weights with marginal benefit density lower than a preset lower limit, the on-chip memory management unit is controlled to release its occupied space, or, if data consistency needs to be maintained, write the relevant data back to the DDR memory; for KV cache pages, the on-chip memory management unit is controlled to perform allocation, retention, marking, or write-back operations based on the current context length and page state. Thus, the evaluation and arbitration results obtained in steps 300 to 450 are ultimately transformed into specific control operations on the on-chip memory management unit and the DDR controller, rather than remaining at the level of abstract calculation results.
[0073] Step 600: Steady-state convergence determination and exit During continuous decoding, as the transport delay correction and KV buffer usage gradually change, the effective overlap gain from adding extra overlap layers may gradually stabilize. Specifically, when the scheduler detects that the effective overlap gain corresponding to the target overlap layer number changes little over several consecutive decoding rounds, it can determine that the system has entered a decoding steady state. In one embodiment, the steady-state convergence condition can be expressed as: in continuous... Within each decoding round, the target overlap layer number The rate of change of the corresponding effective overlapping returns satisfies the following formula: in, The effective overlap reward corresponding to the number of target overlap layers in the current decoding round; The effective overlap gain corresponds to the number of overlapping layers of the same target in the initial round of the steady-state observation window; This is the length of the steady-state observation window; To prevent extremely small positive numbers with a denominator of zero; This is the steady-state convergence threshold; This is a function for taking the absolute value. The threshold parameters mentioned above can be determined by calibration on a specific hardware platform. It should be noted that when... When the denominator of Formula 8 is close to zero, it is almost entirely composed of... The decision, at this point, is correct. The value of is quite sensitive; therefore, optionally, this embodiment can also be supplemented with an absolute change criterion, that is, when the absolute change of the effective overlapping benefit is also lower than the set threshold, it is determined that the decoding steady state has been entered. This auxiliary criterion can be used as a protection condition in engineering implementation without changing the basic technical concept of steady state detection.
[0074] Furthermore, after entering the decoding steady state, the scheduler freezes the current target overlap layer number. Stop full evaluation in each round to reduce scheduling overhead, and release overlapping layers with marginal benefit density below the lower threshold back to the mains. This is used for KV cache or temporary buffers. It should be noted that entering the decoding steady state does not mean permanently fixing the target overlap layer number. Optionally, the scheduler exits the decoding steady state and resumes full evaluation when any of the following exit conditions are detected: the change in the current context length relative to the context length when entering the decoding steady state exceeds a set length change threshold; the number of KV cache pages retained on-chip increases exceeds a set page increase threshold; or the deviation of the measured transport time of the DDR memory from the predicted value exceeds a set deviation threshold. This setting allows the system to reduce overhead when the benefits tend to stabilize, and to promptly resume dynamic adjustment of the overlap layer number when the long context continues to grow, the number of KV cache pages increases, or the DDR channel state changes, avoiding scheduling strategy lag caused by steady-state freezing.
[0075] Step 700: Degradation handling of abnormal operating conditions To improve the robustness of the solution under non-ideal operating conditions, this embodiment further provides several degradation handling methods. Firstly, for sudden drops in DDR bandwidth: when the measured bandwidth... When the bandwidth falls below a set nominal threshold, the scheduler reduces the current target overlap layer number (e.g., halve and round down) and shortens the evaluation period until the measured bandwidth recovers to the preset recovery condition. Furthermore, if the bandwidth remains below the threshold and prefetching tasks repeatedly fail to complete on time, a conservative retention mode is entered, reducing the target overlap layer number to zero or retaining only the data necessary for the upcoming calculation layer, prioritizing the release of on-chip space and reducing failed prefetches. Secondly, regarding PIM calculation time anomalies: when a certain layer... If the value exceeds a set multiple of its recent historical average, the layer is marked as a "computational anomaly layer," and overlapping schemes containing this layer are removed from subsequent candidate evaluations. The layer is restored after the anomaly is resolved. Thirdly, regarding prefetching failures: if the prefetching of data for a certain layer is not completed before the start of the layer's computation and this occurs consecutively for a preset number of times, the layer is marked as a prefetching failure layer. In subsequent decoding rounds, this layer is prohibited from being allocated an advance prefetch depth exceeding one layer's depth. After the set number of rounds or when the DDR measured transfer time returns to normal, the layer is reinstated into the candidate evaluation scope, thus preventing it from permanently losing optimization opportunities. Fourth, regarding KV cache page allocation failure: When the on-chip memory management unit fails to allocate new pages to the KV cache for a preset number of consecutive times, an emergency write-back is triggered. All overlap layer weights and some KV cold pages with low marginal gain density or low access priority are written back to the DDR memory, and the target overlap layer number is forcibly set to zero to enter a conservative retention mode. In conservative retention mode, the scheduler prioritizes ensuring the space required for current decoding calculations and new KV cache writes. Only after the remaining allocable space on the chip is restored to meet the set conditions will it retry performing overlap layer prefetching. It should be noted that the processing parameters mentioned above, such as "halving," "set multiple," "preset number of times," "set ratio," and "set conditions," can be specific values in one implementation and can all be adjusted according to hardware characteristics. This embodiment does not limit this. This degradation process acknowledges that there is a space trade-off between the KV cache and overlap layer weights in extreme scenarios. Its purpose is to reduce the risk of inference interruption, rather than promising to completely eliminate space constraints in all cases.
[0076] Optional evaluation frequency adjustment methods To further reduce runtime evaluation overhead, this embodiment also provides a lookup table evaluation mode and a segmented evaluation mode, which can be selected or combined by those skilled in the art according to the scenario. In the lookup table evaluation mode, a lookup table can be generated offline before deployment based on the number of model layers, hardware configuration, and typical operating state. The lookup table is indexed by the current context length, the current decoding round, and the remaining on-chip allocable space, and the target overlap layer number is output. During runtime, the scheduler directly queries the lookup table to determine the target overlap layer number. When the deviation between the lookup result and the measured benefit or measured transport time exceeds a set threshold, it reverts to the full calculation mode. In the segmented evaluation mode, the decoding process can be divided into a startup segment, a transition segment, and a steady-state segment. The startup segment performs state acquisition and overlap benefit evaluation in each round. The transition segment performs evaluation every set number of rounds. The steady-state segment freezes or exits according to the rules in step 600. It should be noted that neither of the above two modes changes the technical approach of determining the target overlap layer number based on the state during runtime, but rather provides optional implementations for different hardware resources and scheduling overhead requirements.
[0077] III. Device (System) Implementation Examples Based on the same inventive concept as the methods described above, this embodiment also provides a storage scheduling system for the large model decoding stage. For example... Figure 2 As shown, the system includes a scheduler, an in-memory computing PIM chip, a double data rate DDR memory, a DDR controller, and an on-chip memory management unit for the PIM chip. It should be noted that the modules included in the scheduler can be software functional modules, hardware logic modules, or can be implemented jointly by software programs and hardware control units. The scheduler includes the following mutually cooperating modules.
[0078] The status acquisition module is configured to perform the aforementioned step 100 to acquire hardware operating status data including the current context length, the current decoding round, the remaining on-chip allocable space of the PIM chip, the weight size of the layer to be calculated, the KV cache size, the measured handling time of the DDR memory, and the calculation time of the PIM chip.
[0079] The candidate space evaluation module is configured to perform the aforementioned step 200, and based on the remaining on-chip allocable space, the KV cache size and the weight size corresponding to the candidate overlapping layers, to judge the spatial feasibility of multiple preset or dynamically generated candidate overlapping layers, and to select the spatially feasible candidate overlapping layers.
[0080] The overlap benefit evaluation module is configured to perform the aforementioned step 300, update the transport delay correction amount of the current decoding round based on the residual time of the DDR transport that has not yet been completed at the end of the previous decoding round and the idle time between rounds, thereby correcting the calculation window that can be used to hide the DDR transport time, and calculate the effective overlap benefit corresponding to each spatially feasible candidate overlap layer number in combination with the corrected calculation window.
[0081] The arbitration and selection module is configured to perform the aforementioned step 400, calculate the corresponding marginal benefit density based on the effective overlap benefit of each feasible candidate overlap layer in the space, and under the constraint that the KV cache allocation priority is higher than the overlap layer weight retention priority, eliminate candidates with insufficient benefits based on the marginal benefit density and select the target overlap layer from the remaining candidates. In the event of on-chip space contention, the overlap layer weights are released from low to high according to the marginal benefit density to ensure KV cache allocation.
[0082] The scheduling control module is configured to execute the aforementioned step 500, control the on-chip memory management unit to allocate or retain KV cache pages, release or write back the weight space of low-yield overlapping layers according to the target number of overlapping layers, and issue a weight prefetch instruction for the corresponding layer to be calculated to the DDR controller.
[0083] The steady-state detection module is configured to execute the aforementioned step 600. During continuous decoding, when the rate of change of the effective overlap gain corresponding to the target overlap layer number meets the steady-state convergence condition, it determines to enter the decoding steady state, freezes the target overlap layer number to stop the full evaluation in each round and releases the space of the low-gain overlap layer weight, and exits the steady state when the context length, KV cache page number or DDR measured transport time changes to meet the re-evaluation condition.
[0084] Specifically, the modules described above form a closed loop by transmitting state data, candidate overlap layers, effective overlap gains, marginal gain density, and scheduling control instructions. This allows the scheduling results to influence the on-chip space state and DDR transport state of subsequent decoding rounds. Optionally, the scheduler may also include a prefetch control module configured to execute the processing related to prefetch failure marking and prefetch depth limits in step 700. It should be noted that the above module division is only an exemplary logical division and can be merged or split in actual implementation. Any implementation that achieves the above functions falls within the scope of this embodiment.
[0085] IV. Examples of Electronic Devices This embodiment also provides an electronic device, including at least one processor, an in-memory computing PIM chip with on-chip memory space, a double data rate DDR memory, and a memory communicatively connected to the at least one processor. The memory stores computer program instructions, which, when executed by the at least one processor, implement the storage scheduling method for the large model decoding stage as described in any of the above method embodiments. Specifically, the processor can execute the scheduler's state acquisition, candidate space evaluation, overlap gain evaluation, arbitration and selection, scheduling control, and steady-state detection logic; the PIM chip performs corresponding layer calculations and on-chip data access; and the DDR memory moves weights to the PIM on-chip memory or receives write-back data through a DDR controller.
[0086] V. Examples of Computer-Readable Storage Media This embodiment also provides a computer-readable storage medium storing computer program instructions thereon. When executed by a processor, the computer program instructions implement the steps of any of the methods described in the above method embodiments. It should be noted that the computer-readable storage medium can be a non-volatile storage medium or other medium capable of storing program instructions and being read and executed by a processor. Those skilled in the art will understand that all or part of the processes in the above method embodiments can be implemented using hardware related to the computer program instructions, and the program can be stored in a computer-readable storage medium.
[0087] VI. Technical Effects The above embodiments have the following technical effects: Firstly, by introducing a handling delay correction amount Quantifying the DDR transport residue between consecutive decoding rounds helps reduce the risk of overestimating the effective overlap window, thereby reducing the waste of PIM on-chip space caused by blindly retaining invalid overlap layers.
[0088] Secondly, through marginal revenue density By linking the hidden time gain from retaining an extra layer of weights with on-chip space usage, the scheduler can prioritize releasing low-yield overlapping layer weights and retaining more valuable overlapping layers when on-chip space is limited, making the layer number decision under space-constrained conditions discriminative and sortable.
[0089] Third, by placing KV cache page allocation and overlap layer weight retention in the same space arbitration process, and prioritizing KV cache allocation over overlap layer weight retention, the space required for KV cache writing and current computation is prioritized when the KV cache continues to grow, reducing the risk of inference interruption caused by excessive overlap layer weight occupying on-chip space.
[0090] Fourth, by using hysteresis thresholds, prefetch failure layer marking, and abnormal degradation processing, it helps to reduce the additional scheduling overhead caused by small fluctuations in revenue or repeated prefetch failures between adjacent decoding rounds.
[0091] Fifth, by using steady-state convergence determination and multiple exit conditions, it helps to reduce the overhead of full evaluation in each round when the effective overlap benefits tend to stabilize, and restores full evaluation when the context length, KV cache page count or DDR transport status changes, thus taking into account both scheduling overhead and runtime adaptability.
[0092] Furthermore, in short-context scenarios, it helps to utilize the surplus PIM on-chip space to retain more overlapping layers, thereby potentially improving decoding speed; in long-context scenarios, it helps to reduce inefficient on-chip retention and mitigate the risk of space constraints caused by KV cache growth. It should be noted that the above effects are beneficial effects that may be achieved under the corresponding implementation methods. Under extreme conditions, the system may still trigger the aforementioned degradation processing, which has been described accordingly in this embodiment. Therefore, the above effects do not constitute an absolute commitment.
[0093] The following exemplary embodiments further illustrate the present application.
[0094] For example, suppose that the weights of several adjacent layers to be calculated in a certain model are similar in size, and the time required for DDR to transfer the weight of a single layer is about a fraction of the calculation time of a single layer PIM. The previous round of transfer delay correction amount This is approximately a small proportion of the computation time for a single layer. The first step is to select candidates... Estimate the on-chip space requirements using Formula 3, where The first step is to eliminate the cache because the sum of the weights and the current KV cache usage exceeds the total on-chip capacity; the second step is to update according to formula 4. Because this round involves continuous decoding with extremely short intervals between rounds, A slight increase compared to the previous round; the third step is to calculate according to formula 5. and Both are positive and Fourth step, calculate the marginal revenue density of the second layer according to formula 6. If its value is greater than the lower threshold, it is retained; otherwise, it is eliminated. The fifth step is to select the candidate that passes the profit constraint. The largest one is The difference between this number and the target number of layers from the previous round is calculated. If the difference is less than the hysteresis threshold, the number of layers from the previous round is maintained. This example is for illustrative purposes only; specific values can be determined based on the target hardware platform calibration.
[0095] In one implementation, The weight transfer request in the previous decoding round is recorded by the DDR controller as the difference between the completion time of the last weight transfer request in the previous decoding round and the end time of the current round's calculation. The time is calculated by the DDR controller based on the cumulative time during which no valid transfer requests occupy the DDR channel between the end of the previous decoding round and the start of the current decoding round. When the hardware controller cannot directly output the above time, the scheduler can perform software estimation based on the transfer request initiation time, the completion interrupt time, and the layer calculation end time.
[0096] In one implementation, the aforementioned thresholds are not written as fixed constants, but are obtained through offline performance profiling or initial deployment statistics. For example, the marginal revenue density lower limit threshold is measured in bytes per time, used to filter overlapping layers with excessively low on-chip space revenue. A value that is too large is conservative and may prematurely eliminate layers that could bring positive revenue; a value that is too small may retain layers with meager revenue, wasting space. The hysteresis threshold has the same dimension as the effective overlapping revenue, used to suppress frequent switching when adjacent candidate revenues are close. A value that is too large results in sluggish switching and may cause the layer to remain at a suboptimal level for an extended period; a value that is too small weakens the effect of suppressing switching. The steady-state observation window length is measured in rounds, and the steady-state convergence threshold is a dimensionless relative quantity; both are used to determine whether continuous decoding revenue tends to stabilize. The length change threshold, page number growth threshold, and DDR deviation threshold are used to determine whether it is necessary to exit steady-state re-evaluation. The specific value range of each of the above thresholds can be determined based on the target hardware platform calibration.
[0097] In one implementation, the candidate overlap layer set can be generated based on the model layer number, remaining on-chip allocable space, and hardware prefetch granularity. For example, it may include multiple candidate levels such as small, medium, and large. To reduce the number of evaluations, a candidate set with a larger increment step size can also be used. The KV cache is allocated according to the page granularity supported by the on-chip memory management unit. Pages within the current attention window, those with high recent access frequency, or those about to be accessed by subsequent layers are marked as high-priority pages, while pages outside the current attention window or those with low recent access frequency are marked as low-priority pages. When on-chip space is insufficient and the demand for new KV cache pages cannot be met even after releasing all overlap layer weights, low-priority pages are preferentially written back to DDR memory. The specific values of the candidate set and the grading rules do not affect the overall method of selecting the target overlap layer number based on space feasibility, effective overlap benefits, and marginal benefit density during runtime in this application.
[0098] In one implementation, when the DDR bandwidth is lower than a set percentage of the nominal bandwidth, the target overlap layer number is reduced and the evaluation cycle is shortened. Once the bandwidth recovers to a set recovery threshold and continues for several decoding rounds, the normal evaluation cycle is restored. If a layer's prefetch fails to complete before the layer's computation begins multiple times consecutively, the prefetch depth of that layer is limited for several subsequent decoding rounds. If the on-chip memory management unit fails to allocate new pages for the KV cache multiple times consecutively, an emergency write-back is triggered, and a conservative retention mode is entered. Furthermore, the PIM chip or inference runtime system can provide statistics on hierarchical computation time, DDR transfer request start and end times, on-chip memory allocation status, and KV cache page status, which the scheduler updates accordingly. , , , and When real-time statistics are unavailable, historical averages or offline performance profiling results can be used as a substitute, and a full evaluation can be restored when the measured deviation exceeds a threshold. All of the above thresholds can be determined by hardware platform calibration.
[0099] In one implementation, to avoid an undefined state in the scheduling process when all candidate overlap layers are eliminated or fail to meet spatial constraints, the candidate overlap layer set may include zero-layer overlap candidates; or, when no valid candidate overlap layers are found after spatial feasibility and marginal benefit density assessments, the scheduler sets the target overlap layer to 0 and enters a conservative retention mode. In conservative retention mode, the scheduler pauses multi-layer overlap weight retention and prioritizes allocating on-chip space to the data required for current decoding calculations, temporary buffers, and newly added KV cache pages; when the remaining on-chip allocable space, DDR measured transfer time, or KV cache page status recovers to meet the re-evaluation conditions, the scheduler resumes a full evaluation of the candidate overlap layers.
[0100] In one implementation, the transport delay correction amount This is used to conservatively adjust the calculation window that can be used to hide DDR transport time in the current decoding round, and does not imply that the system will necessarily reach a fixed convergence value under all operating conditions. The adjusted calculation window is used when there is no inter-round idle time during continuous decoding and the remnants of previous DDR transport persist. It may gradually decrease, or even drop to 0. At this point, the effective overlap gain corresponding to the number of candidate overlap layers... The corresponding reduction occurs; if the effective overlap gain is lower than the set condition, the scheduler performs degradation processing by eliminating low-gain overlap layers through marginal gain density elimination, releasing the weight of low-gain overlap layers, or setting the target overlap layer number to 0. Therefore, the transport delay correction described in this application does not promise to eliminate all DDR waits, but rather conservatively corrects the selection of the overlap layer number under the constraints of limited on-chip space and the existence of continuous transport residues.
[0101] In one implementation, the marginal revenue density lower limit threshold, hysteresis threshold, steady-state observation window length, steady-state convergence threshold, length change threshold, page number growth threshold, DDR deviation threshold, nominal value ratio threshold, preset number of times, and preset ratio can all be determined based on the offline performance profiling results of the target hardware platform, the initial deployment operation statistics, or the historical average of the most recent decoding rounds. The marginal revenue density lower limit threshold is measured in units of time per byte, used to determine whether the unit on-chip space occupancy is sufficient to obtain enough additional time for handling and hiding. The hysteresis threshold has the same unit as the effective overlap revenue, used to suppress frequent switching when adjacent candidate revenues are close. The steady-state convergence threshold is a dimensionless relative quantity, used to determine whether the effective overlap revenue tends to stabilize during continuous decoding.
[0102] In one implementation, to reduce the impact on parameters when the effective overlap gain is close to zero, steady-state convergence judgment is required. To enhance sensitivity, the scheduler can also incorporate an absolute change criterion for judgment. Specifically, when the relative rate of change satisfies the steady-state convergence condition, and the absolute change of the effective overlap gain corresponding to the target overlap layer number within the steady-state observation window is lower than a set absolute change threshold, the system is determined to enter the decoding steady state; if the two conditions are not met simultaneously, full evaluation continues. This process is used to avoid misjudgments caused by an excessively small denominator when the effective overlap gain is close to zero.
[0103] In normal scenarios, if the remaining on-chip allocable space is sufficient to meet the space requirements for adding new KV cache and candidate overlap layer weights, the scheduler selects the target overlap layer number based on the effective overlap benefit and marginal benefit density, and issues a weight prefetch instruction. In critical scenarios, if the difference in effective overlap benefits between adjacent candidates is less than the hysteresis threshold, the scheduler maintains the target overlap layer number from the previous decoding round to reduce frequent switching; however, the hysteresis rule does not apply when KV cache allocation is insufficient or space constraints are not met. In extreme scenarios, if releasing all overlap layer weights still cannot meet the requirements for adding new KV cache pages, the scheduler prioritizes writing back low-priority KV cold pages and sets the target overlap layer number to 0, re-evaluating the candidate overlap layer number after the on-chip space or DDR transfer status is restored.
[0104] For example, the timing diagram can be drawn with three parallel tracks along the time axis, representing the time blocks for each round of PIM layer-by-layer calculation, the time blocks for DDR layer-by-layer transport, and a "correction window that can be used to hide transport" marked with shade below each round of calculation blocks; as can be seen in the diagram, as the rounds progress, the tail of the transport block remains and encroaches on the head of the next round of calculation blocks, and the shaded window narrows accordingly. A process of gradual accumulation from zero. The causal chain diagram can be drawn from left to right as follows: Correction → Calculation →Calculation Eliminate low-yield layers → Determine under KV priority constraints. The nodes of "space arbitration → issuing prefetch / release / writeback instructions" are marked with arrows to indicate their effects on reducing DDR wait time and reducing on-chip space waste.
[0105] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this application, if it refers to performing an action according to an element, it means performing the action at least according to that element, including two cases: performing the action only according to that element, and performing the action according to that element and other elements. Expressions such as "multiple," "repeatedly," and "various" include two, two times, two kinds, and more than two, more than two times, and more than two kinds.
[0106] All references to this application are considered to be incorporated herein by reference in their entirety so that they may serve as a basis for modifications if necessary. Furthermore, it should be understood that after reading the foregoing disclosure of this application, those skilled in the art can make various alterations or modifications to this application, and these equivalent forms also fall within the scope of protection of this application.
Claims
1. A storage scheduling method for a large model-oriented decoding stage, characterized in that, The method is executed by the scheduler and includes the following steps: Status acquisition steps: Acquire hardware operating status data, which includes the current context length, current decoding round, remaining on-chip allocable space of the PIM chip, weight of the layer to be calculated, KV cache size, measured handling time of DDR memory, and calculation time of the PIM chip. Candidate space evaluation steps: Based on the remaining on-chip allocable space in the hardware operating status data, the KV cache size, and the weight size corresponding to the candidate overlapping layers, the space feasibility of multiple preset candidate overlapping layers is judged, and the space feasible candidate overlapping layers are selected. Overlap benefit evaluation steps: Based on the remaining DDR transport time at the end of the previous decoding round and the idle time between rounds, update the transport delay correction amount for the current decoding round; based on the transport delay correction amount, correct the calculation window in the current decoding round that can be used to hide the DDR memory transport time; combined with the corrected calculation window, calculate the effective overlap benefit corresponding to each of the spatially feasible candidate overlap layers. Arbitration and selection steps: Based on the effective overlap benefits of each of the spatially feasible candidate overlap layers, calculate the corresponding marginal benefit density and select the target overlap layer from the spatially feasible candidate overlap layers; Scheduling control steps: Based on the target number of overlapping layers, control the memory management unit of the PIM chip to allocate or reserve KV cache pages, release or write back the weight space of low-yield overlapping layers, and send the corresponding weight prefetch instruction to the DDR controller.
2. The method according to claim 1, characterized in that, In the overlapping benefit evaluation step: Regarding the current first Each decoding round, the transport delay correction amount Iterative updates are performed according to the following formula: During the first round of decoding ; This is the amount of delay correction for the previous decoding round; The remaining unfinished time for the last layer of the previous decoding round; The calculation from the end of the previous decoding round to the current round... The measured idle time of the DDR channel between the start of each decoding round; The function is used to maximize the value, ensuring that the transport delay correction is not negative; this applies when there is no gap in continuous decoding. In this case, the transport delay correction amount is used to characterize the cumulative encroachment of DDR transport residues on the prefetch window of subsequent rounds in consecutive decoding rounds; For the number of feasible candidate overlapping layers in the space Its corresponding effective overlapping benefits Calculate using the following formula: in, The layer number currently being calculated; The layer offset of the layer to be calculated; For the transport of the DDR memory Time required for layer weights; Calculate the first for the PIM chip Time required for each layer; The function is for finding the minimum value; For summation functions; The corrected calculation window represents the actual calculation time available for parallel processing after the processing delay correction. The effective overlapping benefits Indicates adoption The reduction in total DDR latency due to layer overlap compared to not using overlap.
3. The method according to claim 2, characterized in that, In the arbitration and selection steps: marginal revenue density Calculate using the following formula: in, The number of candidate overlapping layers is Effective overlapping benefits at the same time; For the first The weight of the layer; the marginal benefit density represents the additional transport hiding time gained for each additional unit of on-chip space occupancy; When the marginal revenue density When the marginal revenue density is less than the set lower threshold, the corresponding number of spatially feasible candidate overlapping layers is eliminated. The step of selecting the target overlap layer from the spatially feasible candidate overlap layer number includes: selecting from the candidates that have not been eliminated, such that the effective overlap gain is maximized. The largest candidate is used as the target overlap layer number for the current decoding round. When the difference between the effective overlap benefit of the candidate selected in the current decoding round that maximizes the effective overlap benefit and the effective overlap benefit of the target overlap layer number in the previous decoding round is less than a set hysteresis threshold, the target overlap layer number of the previous decoding round is kept as the target overlap layer number for the current decoding round. This reduces frequent switching between adjacent decoding rounds caused by small fluctuations in benefits and the resulting prefetch cancellation and re-initiation overhead.
4. The method according to claim 1, characterized in that, The arbitration and selection steps further include: under the constraint that the KV cache allocation priority is higher than the overlap layer weight retention priority, eliminating candidate overlap layers with insufficient returns based on the marginal return density, and selecting the target overlap layer from the candidate overlap layers that have not been eliminated; wherein, ensuring that the KV cache allocation priority is higher than the overlap layer weight retention priority includes executing the following spatial arbitration rules: When the remaining allocable space on the chip is less than the space required for the newly added KV cache, the overlapping layer weights are released in order of increasing marginal revenue density until the remaining allocable space on the chip meets the requirements of the newly added KV cache, so as to reduce the risk of KV cache allocation failure. The KV cache is allocated on a page-by-page basis, and when the current page is full, a new page is requested from the memory management unit of the PIM chip; When all overlapping layer weights are released, and the remaining on-chip allocable space is still less than the space required for the newly added KV cache, the least recently used KV cache cold pages are written back to the DDR memory and reloaded from the DDR memory when used in subsequent applications. The KV cache pages are divided into high-priority pages and low-priority pages based on the most recently accessed layer and the current context window position. When it is necessary to write back the KV cache pages, the low-priority pages are written back to the DDR memory first.
5. The method according to claim 1, characterized in that, In the candidate space evaluation step: For any candidate number of overlapping layers Its corresponding space requirements Calculate using the following formula: in, This represents the total KV cache currently occupied on the PIM chip. For those that need to be prefetched The sum of the spatial weights of the layers. Calculate the required temporary buffer space for the current layer. The on-chip protection level is used to cover the newly added KV write volume and the space required for prefetch rollback. When the space requirement The candidate overlap layer number is determined when the on-chip memory capacity of the PIM chip is greater than the total on-chip memory capacity, or when the on-chip protection level is less than the amount of newly added KV data in this round. Spatial constraints are not met.
6. The method according to claim 2, characterized in that, The method further includes a steady-state detection step: during continuous decoding, when the rate of change of effective overlap reward corresponding to the target overlap layer number satisfies the steady-state convergence condition, it is determined to enter the decoding steady state, the target overlap layer number is frozen to stop the full evaluation in each round, and the space of the overlap layer weights with marginal reward density lower than the preset lower limit is released; when the re-evaluation condition is met, the decoding steady state is exited and the full evaluation is resumed.
7. The method according to claim 6, characterized in that, In the steady-state detection step: The steady-state convergence condition for determining whether to enter the decoding steady state is that, in continuous... Within each decoding round, the target overlap layer number The rate of change of the corresponding effective overlap payoff satisfies the following formula: in, The length of the steady-state observation window. To prevent positive numbers with a denominator of zero, The steady-state convergence threshold, The threshold parameter is determined by the specific hardware platform calibration and is used to calculate the absolute value function. The conditions for reassessment include any one of the following: The change in the current context length relative to the context length when entering the decoding steady state exceeds the set length change threshold; The number of pages retained in the on-chip key-value cache has increased beyond the set page increase threshold; The measured transport time of the DDR memory deviated from the predicted value by more than the set deviation threshold.
8. The method according to claim 1, characterized in that, The method also includes a degradation handling step to address abnormal operating conditions: When the measured bandwidth of the DDR memory is lower than the set nominal value ratio threshold, the current target overlap layer number is halved and the evaluation cycle is shortened until the measured bandwidth is restored. If the prefetching of a certain layer's weights is not completed before the start of the layer's calculation and this happens consecutively a preset number of times, then the layer is marked as a prefetching failure layer, and no more than one layer's prefetch depth will be allocated to that layer in subsequent decoding rounds. When the memory management unit of the PIM chip fails to allocate new pages for the KV cache for a preset number of consecutive times, an emergency write-back is triggered. All KV cold pages with the lowest preset proportion of overlap layer weights and marginal benefit densities are simultaneously written back to the DDR memory, and the target overlap layer number is forcibly set to zero to enter the conservative retention mode.
9. A storage scheduling system for the decoding stage of large models, characterized in that, The system includes a scheduler, an in-memory computing PIM chip, a double data rate DDR memory, a DDR controller, and an on-chip memory management unit for the PIM chip. The scheduler includes the following cooperating functional modules: The status acquisition module is configured to acquire hardware operating status data, which includes the current context length, the current decoding round, the remaining on-chip allocable space of the PIM chip, the weight of the layer to be calculated, the KV cache size, the measured handling time of the DDR memory, and the calculation time of the PIM chip. The candidate space evaluation module is configured to judge the space feasibility of multiple preset candidate overlapping layers based on the remaining on-chip allocable space, the KV cache size, and the weight size corresponding to the candidate overlapping layers in the hardware operating status data, and to filter out the space-feasible candidate overlapping layers. The overlap benefit evaluation module is configured to update the transport delay correction amount of the current decoding round based on the residual time of DDR transport that has not yet been completed at the end of the previous decoding round and the idle time between rounds; based on the transport delay correction amount, correct the calculation window in the current decoding round that can be used to hide the DDR memory transport time; and combine the corrected calculation window to calculate the effective overlap benefit corresponding to each of the spatially feasible candidate overlap layers. The arbitration and selection module is configured to calculate the corresponding marginal benefit density and select the target number of overlapping layers from the spatially feasible candidate overlapping layers based on the effective overlapping benefits of each of the spatially feasible candidate overlapping layers. The scheduling control module is configured to control the on-chip memory management unit to perform the allocation or retention of KV cache pages, the release or write-back of the weight space of low-yield overlapping layers, and to issue a weight prefetch instruction for the corresponding layer to be calculated to the DDR controller, based on the target number of overlapping layers.
10. An electronic device, characterized in that, include: At least one processor; The in-memory computing PIM chip has on-chip memory space; Double Data Rate (DDR) memory; and a memory communicatively connected to the at least one processor; The memory stores computer program instructions, which, when executed by the at least one processor, implement the memory scheduling method for the large model decoding stage as described in any one of claims 1 to 8.
11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the steps of the storage scheduling method for the large model decoding stage as described in any one of claims 1 to 8.
