Weight prefetching method for large model inference and electronic device

Abstract

The application is suitable for the field of artificial intelligence technology, and provides a weight pre-fetching method for large model inference and an electronic device, including: acquiring signal sensing information at a current time; the signal sensing information includes temperature information, SLC cache water level information, I / O queue monitoring information, and bandwidth information; the temperature information represents a controller temperature of a flash controller and a packaging temperature of a NAND flash particle; the SLC cache water level information represents a water level of SLC cache usage; the I / O queue monitoring information represents depths of NVMe submission queues and completion queues; and the bandwidth information represents a change condition of bandwidth; according to the signal sensing information, state information of a storage device at the current time is determined; according to the state information of the storage device at the current time, target size information of a pre-fetching window is determined; the pre-fetching window is used to control a quantity of pre-fetched model weights during large model inference; and according to the target size information of the pre-fetching window, model weights are acquired from a NAND flash particle to a preset cache area.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a weight prefetching method and electronic device for large model inference. Background Technology

[0002] In edge device large model inference scenarios, model weights are usually stored in flash memory. During inference, the weights need to be prefetched from flash memory to fast cache layer by layer. The fixed-window prefetching strategy is prone to incomplete prefetching when bandwidth is reduced, thus causing computation wait. When bandwidth is sufficient, it will waste bandwidth to prefetch useless data.

[0003] Therefore, how to adaptively adjust the size of the prefetch window to save bandwidth and improve prefetching efficiency is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a weight prefetching method and electronic device for large model inference, which can adaptively adjust the size of the prefetching window, saving bandwidth and improving prefetching efficiency.

[0005] In a first aspect, embodiments of this application provide a weight prefetching method for large model inference, including: Obtain signal sensing information at the current moment; wherein, the signal sensing information includes at least one of temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information, wherein the temperature information represents the controller temperature of the flash memory controller and the packaging temperature of the NAND flash memory chip, the SLC cache level information represents the level of SLC cache usage, the I / O queue monitoring information represents the depth of the NVMe submission queue and completion queue, and the bandwidth information represents the bandwidth change; Based on the signal sensing information, the state information of the storage device at the current moment is determined; wherein, the state information represents the bandwidth degradation condition of the storage device; Based on the state information of the storage device at the current moment, the target size information of the prefetch window is determined; wherein, the prefetch window is used to control the number of model weights prefetched during large model inference; Based on the target size information of the prefetch window, model weights are obtained from the NAND flash memory chip and stored in a preset cache area.

[0006] Secondly, embodiments of this application provide a weight prefetching device for large model inference, comprising: The information acquisition module is used to acquire signal sensing information at the current moment; wherein, the signal sensing information includes at least one of temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information, wherein the temperature information represents the controller temperature of the flash memory controller and the packaging temperature of the NAND flash memory chip, the SLC cache level information represents the level of SLC cache usage, the I / O queue monitoring information represents the depth of the NVMe submission queue and completion queue, and the bandwidth information represents the bandwidth change. The state determination module is used to determine the state information of the storage device at the current moment based on the signal sensing information; wherein the state information represents the bandwidth degradation condition of the storage device; The window adjustment module is used to determine the target size information of the prefetch window based on the state information of the storage device at the current moment; wherein, the prefetch window is used to control the number of model weights prefetched during large model inference; The weight prefetching module is used to obtain model weights from the NAND flash memory chips and store them in a preset cache area based on the target size information of the prefetching window.

[0007] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the method described in the first aspect above.

[0010] In this embodiment, during large model inference, multiple dimensions of signal sensing information can be acquired at the current moment, including temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information. Based on these four dimensions of signal sensing information, the state information of the storage device at the current moment can be determined, i.e., the bandwidth state of the storage device. Based on different state information, the size of the prefetch window can be adaptively adjusted to obtain the target size information of the prefetch window at the current moment. Then, based on the target size information of the prefetch window, model weights are prefetched from the NAND flash memory chips to the cache, enabling the model weights to be used for large model inference. By combining the four dimensions of information, the bandwidth state is classified, improving the accuracy of state information determination. Targeted adjustment of the window size according to different state categories avoids bandwidth waste and improves the prefetching efficiency of weights, thereby improving the inference efficiency of large models. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating the weight prefetching method for large model inference provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the weight prefetching method for large model inference provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the weight prefetching method for large model inference provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the weight prefetching method for large model inference provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the weight prefetching device for large model inference provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0015] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0017] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0019] The weight prefetching method for large model inference provided in this application can be applied to terminal devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, robots, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of terminal device.

[0020] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows: SLC: Single-Level Cell. It is a type of flash memory cell where each cell stores only 1 bit of data. SLC offers fast read / write speeds and a long lifespan, but it is also expensive and has low capacity density. Therefore, NAND flash memory reserves a small physical space to simulate SLC mode, providing a high-throughput write cache, also known as a pseudo-SLC cache.

[0021] I / O: Input / Output.

[0022] NAND: Short for NAND Flash, also known as NOT-AND Flash. NAND flash memory is a non-volatile flash memory chip, which is the storage medium of SSD solid-state drives. The flash memory controller performs read and write operations on the NAND chips, garbage collection (GC), pseudo-SLC cache management, and reports statuses such as temperature, SLC cache level, and I / O queue depth.

[0023] NVMe: Non-Volatile Memory Express, a non-volatile memory host controller interface specification.

[0024] GC: Garbage Collection.

[0025] SSD: Solid-State Drive.

[0026] In large-model inference scenarios on edge devices, model weights are typically stored in flash memory. During inference, weights need to be prefetched from flash memory to fast cache layer by layer. Flash memory bandwidth fluctuates frequently due to physical factors such as temperature throttling, SLC cache exhaustion, and garbage collection pressure. Fixed-window prefetching strategies can lead to incomplete prefetching and computational waits when bandwidth is low, while wasting bandwidth by prefetching useless data when bandwidth is ample.

[0027] Currently, the exponentially weighted moving average algorithm is widely used for network bandwidth estimation, but it only outputs a single bandwidth estimate and does not distinguish the physical causes of bandwidth degradation. In flash memory scenarios, bandwidth degradation has several different physical mechanisms. For example, thermal throttling degradation is gradual, with the temperature rising gradually over several seconds, leading to a gradual decrease in bandwidth, which can be predicted in advance by observing temperature trends; SLC cache exhaustion degradation is precipitous, with bandwidth dropping sharply when the pseudo-SLC cache is full, with a decrease of over 90%, which is unavoidable but recovers relatively quickly; garbage collection pressure degradation is periodic and related to the write pattern. The recovery characteristics of these three mechanisms are also completely different: thermal throttling recovery requires waiting for the temperature to decrease due to physical heat dissipation, resulting in slow and delayed recovery; SLC cache exhaustion recovery requires waiting for garbage collection to migrate cached data and release space, after which bandwidth can recover relatively quickly; garbage collection pressure recovery occurs naturally after GC is completed. Current methods do not distinguish between these three causes, resulting in degradation and recovery strategies that cannot adapt to the physical characteristics.

[0028] Existing degradation mechanisms are all threshold-triggered, triggering degradation only after bandwidth falls below a preset threshold—a reactive response. For gradual degradation, there's a several-second time window between the start of bandwidth decline and the threshold being breached. However, existing prefetching systems don't utilize this window for preventative contraction, resulting in prefetching not being completed when the threshold is triggered. Furthermore, existing recovery mechanisms use a uniform step size for gradual recovery, without distinguishing the cause of degradation. This could lead to the window being enlarged before bandwidth has truly recovered, causing oscillations, or wasting already recovered bandwidth.

[0029] General Description of Embodiments in this Application According to one embodiment of this application, a weight prefetching method for large model inference is provided to achieve adaptive adjustment of the prefetching window in large model inference scenarios on edge devices.

[0030] Figure 1 This is a flowchart illustrating a weight prefetching method used for large model inference, as shown below. Figure 1 As shown, a weight prefetching method for large model inference according to an embodiment of this application may include: Step 110: Obtain the signal sensing information at the current moment; wherein, the signal sensing information includes at least one of temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information. Temperature information represents the controller temperature of the flash memory controller and the packaging temperature of the NAND flash memory chip. SLC cache level information represents the level of SLC cache usage. I / O queue monitoring information represents the depth of the NVMe submission queue and completion queue. Bandwidth information represents the changes in bandwidth. Step 120: Determine the state information of the storage device at the current moment based on the signal sensing information; wherein, the state information represents the bandwidth degradation condition of the storage device; Step 130: Determine the target size information of the prefetch window based on the state information of the storage device at the current moment; wherein, the prefetch window is used to control the number of model weights prefetched during large model inference; Step 140: Based on the target size information of the prefetch window, obtain the model weights from the NAND flash memory chips and store them in the preset cache area.

[0031] Steps 110-140 are described in detail below.

[0032] In step 110, during the inference process of the large model, signal sensing information in multiple dimensions can be acquired in real time or periodically to achieve multi-signal bandwidth sensing acquisition. Signal sensing information can include temperature information, SLC cache level information, I / O queue monitoring information, bandwidth information, etc. For example, signal acquisition in the above four dimensions can be performed at a fixed inference cycle using a storage bandwidth monitoring unit and a bandwidth sampler. The storage bandwidth monitoring unit can be used to read the temperature register value exposed by the flash controller, the SLC cache usage level, the I / O queue depth, etc., while the bandwidth sampler is used to record the completion time and data volume of each prefetch operation. The temperature register value can refer to temperature information, the SLC cache usage level can refer to SLC cache level information, the I / O queue depth can refer to I / O queue monitoring, and the completion time and data volume of each prefetch operation can refer to bandwidth information.

[0033] Specifically, temperature information can characterize the controller temperature of the flash memory controller and the package temperature of the NAND flash memory chips. This type of temperature information can be derived from the exposed status data of the flash memory controller, reflecting the heat dissipation conditions of the flash memory controller and NAND flash memory chips, providing a basis for identifying heat-related bandwidth degradation. SLC cache level information can characterize the usage level of the pseudo-SLC cache. This level can be obtained based on the correspondence between the occupied capacity of the pseudo-SLC cache and the total capacity of the pseudo-SLC cache, reflecting the degree of pseudo-SLC cache space occupancy, and is used to identify bandwidth fluctuations caused by the pseudo-SLC cache being fully occupied. I / O queue monitoring information can characterize the queue depth of the NVMe submission queue and the completion queue. By observing the combined depths of these two types of queues, it is possible to distinguish between bandwidth fluctuations caused by resource preemption by background tasks within the flash memory and queue changes caused by upper-layer scheduling, providing a basis for identifying bandwidth degradation caused by garbage collection. Bandwidth information can characterize the changes in bandwidth. The original instantaneous bandwidth can be calculated based on the amount of data and the actual time of a single prefetch operation. Then, after smoothing by dual-speed exponential weighted moving average, the corresponding smoothed bandwidth, bandwidth estimate, and trend ratio are obtained. It can reflect both the absolute throughput level of bandwidth and the direction and severity of bandwidth change relative to the long-term steady-state baseline, providing signal support for distinguishing different types of bandwidth degradation trends.

[0034] In step 120, after obtaining the signal sensing information, the state information of the storage device at the current moment can be determined based on the signal sensing information. The state information is used to characterize the bandwidth degradation condition of the storage device, that is, to classify the bandwidth state. This process does not rely solely on a single bandwidth threshold for judgment, but combines signal sensing information from multiple physical dimensions to distinguish and identify the causes of bandwidth decline and the severity of degradation, thereby obtaining different types of bandwidth degradation conditions, and thus corresponding to differentiated processing logic in the subsequent stages.

[0035] For example, it is possible to identify progressive thermal degradation based on temperature-related change trends, to identify sudden cache exhaustion degradation by combining bandwidth mutation characteristics with SLC cache level, to identify degradation caused by periodic background tasks by combining queue fluctuation characteristics, to identify throttling degradation where bandwidth has substantially dropped, and to identify normal operating conditions where the storage device is in a state without obvious degradation.

[0036] In step 130, after determining the current state information of the storage device, the target size information of the prefetch window can be determined based on the state information of the storage device. The prefetch window is used to control the number of model weights read in a single prefetch operation during large model inference. The target size information of the prefetch window can correspond to the data scale of a single prefetch obtained under different operating conditions.

[0037] A decision bus is pre-configured. This decision bus is a lightweight predictor (typically a decision tree or shallow neural network) trained on historical data. Its input consists of features from several past inference cycles (including hit rate, prefetch completion time, remaining queue depth, bandwidth estimate, etc.), and its output is the optimal window size and its confidence level. The predictor is refreshed periodically and continuously optimized using incremental learning. This embodiment does not require rewriting the decision bus; it only requires the decision bus to respond to confidence adjustment signals, i.e., a decreasing trend signal driving a decrease in confidence. The decision bus retains autonomous decision-making power but can respond to system-level confidence adjustment signals. Confidence is an internal parameter of the decision bus (ranging from 0 to 1), representing the bus's confidence in its output window size. The decision bus predicts the optimal window size based on multi-dimensional signals such as historical hit rate, latency, and bandwidth, and outputs the confidence level of this prediction. When the confidence level is high, the predicted value is executed; when the confidence level is low, a conservative value is executed. The sources of confidence can include: (1) statistics based on historical prediction accuracy, that is, the degree of agreement between the actual optimal window and the prediction window in the past N predictions, and a high degree of agreement indicates a high confidence; (2) based on the stability of the input signal, that is, the more stable the input signal, the higher the confidence, and the greater the jitter, the lower the confidence. This embodiment does not specifically limit the working process of the decision bus.

[0038] The method for determining the target size information of the prefetch window can be differentiated according to different state information. When the storage device is in normal operating condition, the optimal window size output by the decision bus can be directly used as the target size information at the current moment, that is, the size of the prefetch window does not change. When a gradual degradation warning condition is identified, the optimal window size output by the decision bus can be shrunk to obtain the target size information of the prefetch window. When a sudden cliff-like degradation condition is identified, a conservative safety lower limit can be directly used as the target size information of the prefetch window. When a degradation condition in which substantial throttling has occurred is identified, a conservative window size can be maintained as the target size information of the prefetch window. When a periodically fluctuating degradation condition is identified, the optimal window size output by the decision bus can be used while applying damping constraints to the feedback adjustment loop.

[0039] The decision bus can also receive externally sent trend-decreasing signals. Based on these signals, it lowers the confidence level of its internal predictions and autonomously outputs a more conservative window size. This is then superimposed with external preventative shrinkage constraints to obtain the target size information for the prefetched window, thus achieving smooth window shrinkage and avoiding drastic window size changes. The trend-decreasing signal is a signal used to indicate a reduction in window size.

[0040] In step 140, after obtaining the target size information of the prefetch window, the model weights of the corresponding scale can be read from the NAND flash memory chip according to the target size information of the prefetch window and moved to the preset cache area for large model inference computing unit to read and use.

[0041] The default high-speed cache area is the host-side memory area. The prefetching of model weights is a read operation and will not occupy pseudo-SLC cache space. However, various bandwidth degradation conditions of the storage device will indirectly affect the completion efficiency of this read operation by preempting hardware resources.

[0042] For example, when the storage device is in a sudden deterioration condition, the target size information of the prefetch window is set to a safe lower limit, and only a subset of high-priority model weights necessary for inference operations are prefetched to ensure that key weights can be prefetched on time and avoid inference operations from stalling.

[0043] This embodiment acquires multi-dimensional signal perception information, identifies bandwidth degradation conditions based on physical causes, determines the target size of the prefetch window that is adapted to the operating conditions, and then completes the prefetching and transfer of model weights to the cache based on the target size information of the prefetch window. It can achieve differentiated prefetching scale control for bandwidth degradation from different physical sources. It can make full use of storage bandwidth when bandwidth is sufficient, and can also avoid the prefetching task from failing to complete on time and causing inference calculation to wait at different stages of bandwidth degradation, thereby improving the overall adaptation effect of weight prefetching in large model inference scenarios of edge devices.

[0044] In this embodiment, during large model inference, multiple dimensions of signal sensing information can be acquired at the current moment, including temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information. Based on these four dimensions of signal sensing information, the state information of the storage device at the current moment can be determined, i.e., the bandwidth state of the storage device. Based on different state information, the size of the prefetch window can be adaptively adjusted to obtain the target size information of the prefetch window at the current moment. Then, based on the target size information of the prefetch window, model weights are prefetched from the NAND flash memory chips to the cache, enabling the model weights to be used for large model inference. By combining the four dimensions of information, the bandwidth state is classified, improving the accuracy of state information determination. Targeted adjustment of the window size according to different state categories avoids bandwidth waste and improves the prefetching efficiency of weights, thereby improving the inference efficiency of large models.

[0045] Figure 2 This is a flowchart illustrating a weight prefetching method used for large model inference, as shown below. Figure 2 As shown, a weight prefetching method for large model inference according to an embodiment of this application may include: Step 210: Obtain the amount of data and time consumption information of the model weights prefetched in the previous time step; Step 220: Based on the amount of model weight data and time consumption information pre-fetched in the previous time step, determine the fast line value and slow line value based on the preset dual-speed exponential weighted moving average smoother; wherein, the fast line value represents the bandwidth value of the fast line output of the preset dual-speed exponential weighted moving average smoother, and the slow line value represents the bandwidth value of the slow line output of the preset dual-speed exponential weighted moving average smoother. Step 230: Determine the bandwidth estimate and trend ratio based on the fast line value and the slow line value.

[0046] Steps 210-230 are described in detail below.

[0047] In step 210, the bandwidth information may include a bandwidth estimate and a trend ratio. The bandwidth estimate characterizes the estimated effective storage throughput of the storage device, reflecting the stable data read / write throughput that the storage device can provide from the perspective of upper-layer large-scale model inference services. The trend ratio characterizes the rise and fall trend of bandwidth. It is a relative indicator obtained after normalization and can reflect the direction and degree of deviation of bandwidth from the long-term steady-state baseline. It can identify the early trend of accelerated bandwidth change before the absolute value of bandwidth has dropped significantly, providing signal support for preventive degradation identification.

[0048] Both the bandwidth estimate and the trend ratio can be calculated from the fast line value and the slow line value. The fast line value and the slow line value are derived from the dual-speed exponentially weighted moving average smoother. The dual-speed exponentially weighted moving average smoother can filter noise from the raw bandwidth sampling data generated by the prefetch operation and eliminate the instantaneous bandwidth jitter interference caused by a single prefetch operation.

[0049] When determining bandwidth information, the amount of model weight data and the time consumption information prefetched in the previous time step can be obtained first. The amount of model weight data prefetched in the previous time step represents the scale of data that needs to be read from the NAND flash memory chip in a single prefetch operation, and the time consumption information represents the time elapsed from the issuance of the prefetch instruction to the completion of the prefetch operation. These two types of information can be recorded by the bandwidth sampler after each prefetch operation is completed.

[0050] Based on the amount of model weight data and time consumption information pre-fetched from the previous moment, the fast and slow line values ​​are determined using a pre-defined dual-speed exponentially weighted moving average smoother. The fast line value is the bandwidth value output by the fast path within the dual-speed exponentially weighted moving average smoother. The fast path can be configured with a relatively short memory period, exhibiting high sensitivity to recent bandwidth fluctuations and facilitating the capture of sudden bandwidth drops. The slow line value is the bandwidth value output by the slow path within the dual-speed exponentially weighted moving average smoother. The slow path can be configured with a relatively long memory period, referencing bandwidth sampling results over a longer historical period, thus characterizing the long-term steady-state baseline of the bandwidth and reducing interference from occasional instantaneous jitter in bandwidth determination. The fast and slow paths can receive the same set of instantaneous bandwidth input data, independently completing smoothing calculations without modifying each other's internal computational state. This embodiment does not specifically limit the calculation process of the dual-speed exponentially weighted moving average smoother.

[0051] After obtaining the fast and slow line values, the bandwidth estimate and trend ratio can be determined based on the calculation results of the fast and slow line values. The bandwidth estimate can be determined by combining the values ​​of the fast and slow line values, thereby achieving the effect of the bandwidth estimate quickly following the bandwidth decline, while not rapidly rising with a short-term bandwidth surge. The trend ratio can be obtained based on the deviation between the fast and slow line values, and normalized using the slow line value as a benchmark. For example, the minimum value between the fast and slow line values ​​is determined as the bandwidth estimate; the absolute value of the difference between the fast and slow line values ​​is determined, and the trend ratio is obtained based on the absolute value of the difference and the slow line value. In other words, the minimum value of the two lines is taken as the current bandwidth estimate, realizing an estimation strategy of rapid decline and slow rise. At the same time, a trend signal is calculated: the difference between the fast and slow line values ​​is divided by the slow line value to obtain the normalized trend ratio. A positive and increasing trend ratio indicates an increase in bandwidth, while a negative and increasing absolute value indicates an acceleration in the bandwidth decline.

[0052] In this embodiment, by collecting the data volume and time information of the prefetch operation, the fast line value and slow line value are obtained by using a dual-speed exponential weighted moving average smoother, and then further calculated to generate the bandwidth estimate and trend ratio. This allows for the simultaneous acquisition of the absolute bandwidth throughput estimate and the relative bandwidth change trend, filtering out random noise caused by a single prefetch, and providing a reliable bandwidth dimension signal for identifying bandwidth degradation conditions caused by different reasons.

[0053] In one implementation, based on the amount of model weight data and the time consumption information prefetched at the previous time step, and using a preset dual-speed exponentially weighted moving average smoother, the fast line value and the slow line value are determined, including: determining the instantaneous bandwidth value based on the amount of model weight data and the time consumption information prefetched at the previous time step; wherein the instantaneous bandwidth value represents the actual storage throughput capacity; inputting the instantaneous bandwidth value into the fast line of the preset dual-speed exponentially weighted moving average smoother to obtain the fast line value, and inputting the instantaneous bandwidth value into the slow line of the preset dual-speed exponentially weighted moving average smoother to obtain the slow line value.

[0054] Specifically, the instantaneous bandwidth value characterizes the actual storage throughput observed in a single prefetch operation, reflecting the true throughput performance of the storage device perceived by the upper-layer large-model inference service during a single prefetch interaction. The instantaneous bandwidth value can serve as the raw input to a dual-speed exponentially weighted moving average smoother; the iterative updates of both the fast and slow line values ​​depend on the instantaneous bandwidth value. The instantaneous bandwidth value can be obtained by calculating the amount of data prefetched and the time consumption information; for example, the instantaneous bandwidth value can be equal to the amount of data prefetched this time divided by the actual time consumption of this prefetch.

[0055] After obtaining the instantaneous bandwidth value, it can be input into the fast and slow lines of a preset dual-speed exponentially weighted moving average smoother, yielding the corresponding fast and slow line values. The half-life of the fast line is set to two to three sampling periods (sensitive to recent changes), while the half-life of the slow line is set to eight to ten sampling periods (reflecting long-term trends). The fast and slow lines of the dual-speed exponentially weighted moving average smoother can be calculated in parallel. Each time an instantaneous bandwidth value is obtained, the states of both the fast and slow lines are updated synchronously, outputting the updated fast and slow line values ​​respectively. Different half-life parameters can be configured for the fast and slow lines to differentiate their response characteristics. The half-life parameters can be adjusted to suit the storage device hardware characteristics and the operational characteristics of inference services.

[0056] In this embodiment, the instantaneous bandwidth value is calculated by utilizing the amount of data and time information of a single prefetch. The instantaneous bandwidth value is then fed in parallel into the fast and slow lines of the dual-speed exponential weighted moving average smoother to complete smoothing iterations. Based on the original single prefetch sampling data, the fast line value, which is sensitive to bandwidth abrupt changes, and the slow line value, which represents the long-term steady state, are obtained respectively, providing a smoothed basic bandwidth signal for the subsequent calculation of bandwidth estimates and trend ratios.

[0057] For temperature information acquisition, temperature values ​​exposed by the flash controller via SMART (Self-Monitoring, Analysis and Reporting Technology) or NVMe log pages can be read, including both controller temperature and NAND package temperature readings. These temperature values ​​are then fed into an independent exponentially weighted moving average smoother to obtain smoothed temperature values. Further temperature trend calculations are performed; for example, the first difference of the smoothed temperature values ​​over multiple consecutive sampling periods can be calculated to represent the rate of temperature change, and the second difference can represent the temperature acceleration. A temperature acceleration greater than zero indicates that the temperature is rising rapidly, suggesting that thermal throttling is imminent, even if the current temperature has not yet reached the preset throttling threshold.

[0058] To collect SLC cache water level information, a write volume accumulation counter can be maintained, and the current SLC cache usage level can be estimated based on the nominal pseudo-SLC cache capacity of the storage device. An SLC water level exceeding 80% of a preset value means that the next large block prefetch may trigger a sudden drop in SLC cache exhaustion bandwidth.

[0059] For collecting I / O queue monitoring information, the depths of the NVMe commit queue and completion queue can be read. An abnormally high queue depth (more than twice the normal value) accompanied by periodic fluctuations is a typical signal of garbage collection pressure.

[0060] In this embodiment, during large model inference, multiple dimensions of signal sensing information can be acquired at the current moment, including temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information. Based on these four dimensions of signal sensing information, the state information of the storage device at the current moment can be determined, i.e., the bandwidth state of the storage device. Based on different state information, the size of the prefetch window can be adaptively adjusted to obtain the target size information of the prefetch window at the current moment. Then, based on the target size information of the prefetch window, model weights are prefetched from the NAND flash memory chips to the cache, enabling the model weights to be used for large model inference. By combining the four dimensions of information, the bandwidth state is classified, improving the accuracy of state information determination. Targeted adjustment of the window size according to different state categories avoids bandwidth waste and improves the prefetching efficiency of weights, thereby improving the inference efficiency of large models.

[0061] Figure 3 This is a flowchart illustrating a weight prefetching method used for large model inference, as shown below. Figure 3 As shown, a weight prefetching method for large model inference according to an embodiment of this application may include: Step 310: Obtain the signal sensing information at the current moment; wherein, the signal sensing information includes at least one of temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information. Temperature information represents the controller temperature of the flash memory controller and the packaging temperature of the NAND flash memory chip. SLC cache level information represents the level of SLC cache usage. I / O queue monitoring information represents the depth of the NVMe submission queue and completion queue. Bandwidth information represents the changes in bandwidth. Step 320: If the bandwidth estimate is within a preset bandwidth range and the trend ratio is within a preset trend ratio range, then the storage device's current state is determined to be in a throttled state; wherein, the throttled state indicates that the size information of the prefetch window has been adjusted downwards. Step 330: If the bandwidth estimate is not within the preset bandwidth range, or the trend ratio is not within the preset trend ratio range, then determine the decrease amplitude of the fast line value within the preset time period; if the decrease amplitude is within the preset amplitude range, and the SLC cache level information exceeds the preset level threshold, then determine that the storage device's status information at the current moment is the SLC cache exhaustion state; wherein, the SLC cache exhaustion state indicates that the available space of the pseudo-SLC cache is filled. Step 340: If the decrease value is not within the preset range, or the SLC cache level information does not exceed the preset level threshold, then when the temperature acceleration is greater than the preset temperature threshold and the trend ratio is negative, the storage device's status information at the current moment is determined to be a thermal risk warning state; wherein, the thermal risk warning state indicates that the bandwidth will decrease due to thermal throttling. Step 350: If the temperature acceleration is not greater than the preset temperature threshold, or the trend ratio is not negative, then when the I / O queue monitoring information meets the first preset condition and the bandwidth estimate meets the second preset condition, the storage device's status information at the current moment is determined to be the GC pressure state; wherein, the GC pressure state characterizes the periodic bandwidth jitter caused by garbage collection preempting bandwidth; Step 360: Determine the target size information of the prefetch window based on the state information of the storage device at the current moment; wherein, the prefetch window is used to control the number of model weights prefetched during large model inference; Step 370: Based on the target size information of the prefetch window, obtain the model weights from the NAND flash memory chips and store them in the preset cache area.

[0062] Steps 310, 360 and 370 have been described in detail above and will not be repeated here. Steps 320-350 will be described in detail below.

[0063] In step 320, the signal sensing information output in step 310 is read, and state classification is performed. This embodiment does not simply determine whether to degrade based on whether the bandwidth value is lower than a threshold, but rather determines the cause and severity of the bandwidth decrease based on the combination of multiple physical signals, classifying it into four states, with different protection strategies corresponding to different states.

[0064] The bandwidth information may include bandwidth estimates and trend ratios. When determining the status information, the system first uses the bandwidth estimates and trend ratios to determine whether the current state is one of throttling. A throttling state indicates that the prefetch window has been degraded in this state, and the system enters the prefetch window recovery monitoring process.

[0065] In other words, the throttling state indicates that the size of the prefetch window has been reduced, meaning that the storage device has experienced a substantial bandwidth drop and is no longer suitable for performing prefetch operations of normal scale. After entering the throttling state, the subsequent process will no longer execute the relevant logic for preventive degradation, but will instead enter the recovery monitoring process, continuously monitoring whether the bandwidth has the conditions to recover, and waiting for the storage device's operating condition to improve before executing the corresponding recovery steps.

[0066] The logic for determining whether a throttling state has been entered can be as follows: determine the bandwidth range in which the bandwidth estimate falls and the trend ratio range in which it falls. If the bandwidth estimate falls within the corresponding bandwidth range, it means that the actual throughput capacity of the storage device has significantly decreased. If the trend ratio falls within the corresponding trend ratio range, it can cover two scenarios: either the bandwidth is still decreasing, or the bandwidth has stabilized after decreasing. This helps identify whether the storage device has entered a throttling state.

[0067] For example, the preset bandwidth range corresponding to the bandwidth estimate can be set to a low range relative to the device's normal throughput level, and the preset trend ratio range corresponding to the trend ratio can include the value range corresponding to a continuous decline in bandwidth, as well as the steady-state value range where the fast and slow lines converge after the bandwidth decline is complete. When the bandwidth estimate and trend ratio in the signal sensing information meet the corresponding range conditions, the storage device is determined to have entered a throttling state. For example, if the bandwidth estimate has dropped to below 50% of the preset normal range, and the trend ratio is negative (still decreasing) or trending towards stability (has reached the bottom), it is determined to be in a throttling state. Trending towards stability means that the trend ratio is close to 0.

[0068] In this embodiment, by combining the absolute range judgment of the bandwidth estimate with the relative situation judgment of the trend ratio, the throttling state can be identified. This can simultaneously identify the operating conditions where the bandwidth is still deteriorating and the bandwidth has stabilized after a drop. After the storage device experiences substantial bandwidth degradation, the throttling state can be identified in a timely manner, and the process can be switched to the recovery monitoring process to avoid the inference calculation waiting caused by continuing to use the normal-sized prefetch window.

[0069] A significantly positive and increasing trend ratio indicates that bandwidth is accelerating upwards; a significantly negative trend ratio with an increasing absolute value indicates that bandwidth is accelerating downwards; a trend ratio close to 0 indicates that the fast and slow lines are converging, and bandwidth is in a steady state (whether stable at a high or low level). When determining if bandwidth has been throttled, use a negative trend ratio (still decreasing) or a trending stable ratio (trend ratio close to 0, having reached the bottom). Both of these indicate that bandwidth is still deteriorating or has just bottomed out and has not yet rebounded, and the branch indicating the throttling state should be entered.

[0070] In step 330, the bandwidth information may include the fast line value. If the throttling condition is not met—that is, the estimated bandwidth does not fall within a preset bandwidth range, or the trend ratio does not fall within a preset trend ratio range—the decrease in the fast line value over a preset time period can be further determined. The decrease in the fast line value over the preset time period characterizes the change in bandwidth level over a continuous time period. This decrease is used to characterize whether a sudden drop in bandwidth occurs within a short period, thus distinguishing between gradual bandwidth decrease and sudden, precipitous bandwidth decrease. The precondition for this judgment branch is that the storage device has not yet entered a throttling state, and the bandwidth has not yet dropped to a low steady state. At this time, the focus is on investigating whether a sudden bandwidth degradation caused by a pseudo-SLC cache full has occurred.

[0071] When the decrease in the fast line value falls within a preset range, and the SLC cache level exceeds a preset threshold, the storage device is determined to be in an SLC cache exhaustion state at the current moment. An SLC cache exhaustion state indicates that the available space of the pseudo-SLC cache has been filled with written data. After the pseudo-SLC cache is exhausted, new data needs to be written directly to the low-speed flash memory cells, causing a sudden degradation in the storage device's bandwidth. Relying solely on a short-term, significant bandwidth drop is insufficient to pinpoint the source of the fault; other hardware factors can also cause short-term bandwidth drops. Similarly, a high SLC cache level alone cannot confirm a sudden bandwidth change. Therefore, this embodiment uses both the decrease in the fast line value and the SLC cache level information as conditions for joint determination, eliminating the risk of misjudgment due to a single condition.

[0072] Once the SLC cache exhaustion state is identified, a corresponding mutation flag can be generated. This mutation flag can then be used to trigger the freeze operation of the feedback control module. The feedback control module is used to increase the size of the prefetch window during shrinking, while the freeze operation prevents the prefetch window from increasing in size during shrinking. For example, a mutation flag of 1 indicates that the freeze operation of the feedback control module is triggered; a mutation flag of 0 indicates that the freeze operation of the feedback control module is lifted.

[0073] This embodiment first determines whether the bandwidth is already throttled. If not, it then determines whether the SLC cache is exhausted. If the bandwidth drops by more than 50% (a precipitous drop) within a sampling period, and the SLC level exceeds 80% of the preset normal level, then the SLC cache is determined to be exhausted. This is a sudden drop that cannot be prevented in time. The prefetch window can be immediately reduced to a safe lower limit (prefetching only the highest priority weight), and the mutation flag is set while the feedback correction is frozen. Setting the mutation flag triggers the freeze operation of the feedback control module.

[0074] In this embodiment, by combining the decrease amplitude of the fast line value within a preset time period with the SLC cache level information to jointly determine the SLC cache exhaustion state, the cliff-like bandwidth degradation caused by the pseudo-full SLC cache can be identified, avoiding the misidentification problem caused by a single signal judgment, and providing a reliable state identification result for subsequent targeted sudden degradation processing.

[0075] In step 340, the temperature information may include temperature acceleration. Temperature acceleration is a physical characteristic quantity obtained by performing a second-order difference operation on the temperature smoothing value. It can characterize that the rate of temperature change itself is still changing and can reflect the trend of accelerated temperature rise. Temperature acceleration can capture early signs of accelerated hardware heating before actual thermal throttling causes a bandwidth drop. It can be combined with trend ratio to complete the judgment of thermal risk warning status, that is, to determine whether thermal throttling will occur.

[0076] If the conditions for determining SLC cache exhaustion are not met—that is, the decrease in the fast line value does not fall within the preset range, or the SLC cache level information does not exceed the preset threshold—further assessment of the thermal risk warning state can be conducted. The thermal risk warning state indicates that the storage device's bandwidth will decrease due to thermal throttling. This state is a warning condition preceding degradation; at this point, the bandwidth has not yet experienced a substantial drop, but the hardware has already presented the preconditions that will lead to subsequent bandwidth degradation. After identifying the thermal risk warning state, a preventative degradation process can be triggered, rather than waiting for the bandwidth to drop before taking reactive action. The determination of the thermal risk warning state considers both temperature acceleration and trend ratio. A temperature acceleration greater than the preset temperature threshold indicates that the hardware temperature is accelerating upwards, while a negative trend ratio indicates that the bandwidth has shown a downward trend. Only when both conditions are met is the thermal risk warning state declared.

[0077] A negative trend ratio indicates that the fast line value is smaller than the slow line value, and the short-term bandwidth performance is already worse than the long-term steady-state baseline. Combining the conditions of accelerating temperature rise and bandwidth decline as constraints can reduce misjudgments caused by a simple temperature increase or a small bandwidth fluctuation. The thermal risk warning state will not set the abrupt change flag or freeze the feedback control module; instead, it applies soft constraints to the decision bus by issuing a trend decline signal, thus distinguishing it from the hard degradation logic of the SLC cache exhaustion state.

[0078] In this embodiment, after determining that the non-SLC depletion state is not in effect, a thermal risk warning is issued. For example, if the temperature acceleration is greater than zero (temperature is accelerating upward) and the trend ratio is negative (bandwidth has begun to decrease), a thermal risk warning is issued. This is an early signal of a gradual decline; the bandwidth has not yet decreased significantly but will soon decrease due to thermal throttling. For a thermal risk warning state, a preventative degradation can be triggered, i.e., the prefetch window can be shrunk to 70% of its normal value, the abrupt change flag is not set, but a trend decline signal is sent to the decision bus to reduce the confidence level.

[0079] In this embodiment, the hardware characteristics of accelerated temperature rise are determined by temperature acceleration, and the trend ratio representing the downward trend of bandwidth is used to jointly identify the thermal risk warning state. The warning identification can be completed before thermal throttling actually causes a significant degradation of bandwidth, providing a trigger condition for the preventive smooth contraction of the prefetch window and reducing the probability of subsequent prefetch tasks failing to be completed on time.

[0080] In step 350, the GC pressure state characterizes the periodic bandwidth jitter caused by garbage collection preempting bandwidth. Garbage collection is a data sorting task executed in the background of the flash controller. During the garbage collection process, it will occupy the read and write resources of the flash hardware, thereby causing periodic fluctuations in the bandwidth of the storage device.

[0081] The GC stress state differs from the throttling state, SLC cache exhaustion state, and thermal risk warning state. Under the GC stress state, the overall bandwidth does not experience a substantial drop, but only exhibits periodic disturbances. After this state is identified, it does not trigger the prefetch window's active degradation. Instead, it sends a corresponding signal to the feedback control module to change the adjustment characteristics within the feedback control module. This avoids the feedback control module misjudging the periodic jitter as a change in bandwidth trend. The GC stress state identification can continue after excluding the thermal risk warning state, thus improving the complete classification logic for multiple bandwidth degradation conditions.

[0082] If the conditions for determining a thermal risk warning state are not met—that is, the temperature acceleration is not greater than a preset temperature threshold, or the trend ratio is not negative—then a GC stress state determination can be further carried out. This prerequisite indicates that the storage device does not experience thermal risks due to accelerated temperature increases, nor does the bandwidth show a downward trend. In this case, I / O queue monitoring information and bandwidth estimates are further examined. A first and second preset condition are pre-set. If the I / O queue monitoring information meets the first preset condition, and the bandwidth estimate meets the second preset condition, then the storage device's status is determined to be GC stress. The first preset condition mainly targets I / O queue monitoring information to identify abnormally high queue depths accompanied by periodic fluctuations. The second preset condition mainly targets bandwidth estimates to confirm that the bandwidth estimate is still within the normal range, with only periodic fluctuations. These two conditions work together to avoid misidentifying queue anomalies caused by upper-layer service scheduling as bandwidth disturbances caused by garbage collection.

[0083] For example, I / O queue monitoring information can be judged by combining the queue depth of the NVMe commit queue and the completion queue. If the queue depth of both the commit queue and the completion queue is high and shows periodic fluctuations, it corresponds to the first preset condition. If the bandwidth estimate is maintained within the normal working range and only fluctuates periodically, it corresponds to the second preset condition. That is, the first preset condition corresponds to the I / O queue depth and fluctuation, and the second preset condition corresponds to the fluctuation of the bandwidth estimate. I / O queue monitoring information includes the read NVMe commit queue (the number of requests that have been issued but not yet processed by the device) and the completion queue (the number of requests that have been completed by the device but not yet processed by the host for the completion callback). There are several typical characteristics of abnormally high queue depth: (1) The commit queue depth spikes while the completion queue depth is stable, indicating that the device cannot keep up with the commit rate, which may be due to busy GC or hot throttling on the device side; (2) The commit and completion queue depths are both high and fluctuate periodically, indicating that GC is periodically seizing bandwidth; (3) The completion queue depth spikes while the commit queue is stable, which may be due to upper-layer scheduling problems and is not related to the decrease in flash bandwidth. Only by examining both queue depths together can the physical cause of bandwidth degradation be identified; looking at only one can easily lead to misdiagnosis.

[0084] In other words, if the conditions of throttling, SLC cache exhaustion, and thermal risk warning are not met, then it is determined whether it is a GC pressure fluctuation state. If the I / O queue depth is abnormally high and has periodic fluctuation characteristics, and the bandwidth estimate fluctuates periodically within the normal range, then it is determined to be a GC pressure state. In this state, no preventive degradation is performed, but a GC fluctuation signal is sent to the feedback control module so that the feedback controller can increase the integral term damping to avoid overreacting to periodic fluctuations.

[0085] That is, after the GC pressure state is identified, the abrupt change flag is not set, and the prefetch window still uses the window size output by the decision bus to perform the prefetch operation. Only the integral term of the feedback control module is subject to damping constraints to suppress the feedback control module from over-adjusting in response to periodic jitter. In other words, the integral time is increased and the integral weight is reduced to avoid the feedback control module misinterpreting periodic fluctuations as trend changes and overreacting (such as incorrectly shrinking the window).

[0086] In this embodiment, the GC pressure state is identified by combining the first preset condition of the I / O queue monitoring information with the second preset condition of the bandwidth estimate. This can identify the periodic bandwidth jitter caused by the garbage collection task preempting hardware resources. Without shrinking the prefetch window, a damping constraint is applied to the feedback control module to avoid the feedback control module making incorrect adjustments to the periodic jitter, thus ensuring that the prefetch process can make full use of the available bandwidth of the storage device.

[0087] It is worth noting that the mutation flag is only set in the SLC cache exhaustion state and the throttling state; it is not set in the thermal risk warning state and the GC pressure state. This is because the thermal risk warning state and the GC pressure state do not need to skip the normal decision-making of the decision bus, but rather affect the parameters of the decision bus through signal injection.

[0088] In this embodiment, during large model inference, multiple dimensions of signal sensing information can be acquired at the current moment, including temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information. Based on these four dimensions of signal sensing information, the state information of the storage device at the current moment can be determined, i.e., the bandwidth state of the storage device. Based on different state information, the size of the prefetch window can be adaptively adjusted to obtain the target size information of the prefetch window at the current moment. Then, based on the target size information of the prefetch window, model weights are prefetched from the NAND flash memory chips to the cache, enabling the model weights to be used for large model inference. By combining the four dimensions of information, the bandwidth state is classified, improving the accuracy of state information determination. Targeted adjustment of the window size according to different state categories avoids bandwidth waste and improves the prefetching efficiency of weights, thereby improving the inference efficiency of large models.

[0089] Figure 4 This is a flowchart illustrating a weight prefetching method used for large model inference, as shown below. Figure 4 As shown, a weight prefetching method for large model inference according to an embodiment of this application may include: Step 410: Obtain the signal sensing information at the current moment; wherein, the signal sensing information includes at least one of temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information. Temperature information represents the controller temperature of the flash memory controller and the packaging temperature of the NAND flash memory chip. SLC cache level information represents the level of SLC cache usage. I / O queue monitoring information represents the depth of the NVMe submission queue and completion queue. Bandwidth information represents the changes in bandwidth. Step 420: Determine the state information of the storage device at the current moment based on the signal sensing information; wherein, the state information represents the bandwidth degradation condition of the storage device; Step 430: Determine the target size information of the prefetch window based on the state information of the storage device at the current moment; wherein, the prefetch window is used to control the number of model weights prefetched during large model inference; Step 440: Based on the target size information of the prefetch window, obtain the model weights from the NAND flash memory chips and store them in the preset cache area; Step 450: Determine the target monitoring information based on the status information of the storage device at the current moment; wherein, the target monitoring information represents the signal sensing information used to confirm the status information at the current moment; Step 460: If the target monitoring information meets the preset window recovery conditions, then determine the window recovery coefficient corresponding to the state information at the current moment according to the preset correlation relationship; wherein, the preset correlation relationship represents the correlation relationship between the state information and the window recovery coefficient. Step 470: Adjust the target size information of the prefetched window according to the window recovery coefficient corresponding to the state information at the current time, and obtain the recovered size information of the prefetched window.

[0090] Steps 410, 420, and 440 have been described in detail above and will not be repeated here. Steps 430 and 450-470 will be described in detail below.

[0091] In step 430, different window shrinking strategies can be executed for different state information. For example, the window shrinking magnitude corresponding to different state information can be different. If the state information is not a throttling state, SLC cache exhaustion state, hot risk warning state, or GC pressure state, it is determined to be a normal state. At this time, the bandwidth is stable, and the window size output by the decision bus is the target size information. That is to say, for the normal state, the window size output by the decision bus is directly adopted without any adjustment, the prefetching operation is performed, and the prefetching completion status and hit rate are monitored, and the results are sent to the feedback control module.

[0092] In one embodiment, determining the target size information of the prefetch window based on the current state information of the storage device includes: if the current state information of the storage device is a thermal risk warning state, then determining the initial adjustment size information of the prefetch window based on the current size information of the prefetch window; wherein, the thermal risk warning state indicates that the bandwidth will decrease due to thermal throttling; the initial adjustment size information indicates the size information after the prefetch window is initially shrunk; and determining the target size information of the prefetch window based on the initial adjustment size information and a preset window shrinkage coefficient.

[0093] Specifically, the thermal risk warning state indicates that bandwidth will decrease due to thermal throttling. This state is a warning condition before bandwidth degradation occurs; at this point, no substantial bandwidth drop has occurred. Therefore, instead of a direct forced jump to the conservative lower limit degradation method, a layered smooth contraction processing logic is adopted. After confirming the thermal risk warning state, a trend decline signal can be sent to the decision bus. That is, the identification result of the thermal risk warning state can trigger the decision bus to lower its confidence level, thereby reducing the size of the prefetch window. The current size information of the prefetch window can be the window size output by the decision bus before responding to the trend decline signal. After confirming the state information as a thermal risk warning state, the decision bus lowers the confidence level, causing the decision bus to output a smaller window based on the current size information, thus obtaining the initial adjustment size information.

[0094] A window shrinkage factor is preset; for example, the preset window shrinkage factor could be 0.7. The initial size information output by the decision bus is multiplied by the preset window shrinkage factor to obtain the target size information.

[0095] First, the initial adjustment size of the prefetch window is determined based on its current size. This initial adjustment size represents the size of the prefetch window after initial shrinkage adjustment. Upon receiving a trend-decreasing signal, the decision bus lowers its internal confidence level. After the confidence level is lowered, the decision bus spontaneously outputs a more conservative window output result based on its own prediction logic—that is, the initial adjustment size of the prefetch window. This process is a soft shrinkage driven by the decision bus's internal logic and will not cause drastic changes in the window value. Even without an externally applied window shrinkage coefficient, a certain degree of window shrinkage can be achieved solely by lowering the decision bus's confidence weight. The preset window shrinkage coefficient serves as a second layer of safety constraint, which can be used to address situations where the decision bus response is untimely or insufficient.

[0096] After obtaining the initial adjustment size information, the target size information of the prefetched window can be further calculated by combining the preset window shrinkage coefficient. The preset window shrinkage coefficient serves as an externally applied safety constraint and is superimposed on the initial adjustment size information obtained by the endogenous shrinkage of the decision bus, forming a dual shrinkage protection mechanism.

[0097] In this embodiment, the initial size information of the prefetch window is obtained by lowering the confidence level of the decision bus. Then, the target size information of the prefetch window is determined by combining the preset window shrinkage coefficient. This can achieve layered smooth window shrinkage. It can make conservative output by utilizing the prediction logic of the decision bus itself, and increase the safety margin by using the external shrinkage coefficient to avoid sudden changes in the window. The prefetch size is adjusted and adapted before the actual hot throttling occurs, reducing the possibility of waiting during inference calculation.

[0098] In one implementation, determining the target size information of the prefetch window based on the current state information of the storage device includes: if the current state information of the storage device is an SLC cache exhaustion state, then determining the preset minimum size information as the target size information of the prefetch window; wherein, the SLC cache exhaustion state indicates that the bandwidth will decrease due to thermal throttling; the initial size information indicates that the available space of the pseudo SLC cache is filled.

[0099] Specifically, the SLC cache exhaustion state indicates that the available space of the pseudo-SLC cache is full. When the available space of the pseudo-SLC cache is exhausted, write operations need to directly operate on the native flash memory storage cells, which will cause a sudden and significant degradation of the storage device's bandwidth. The SLC cache exhaustion state is a sudden failure condition, and there is no processing time for the system to perform smooth preventive shrinkage. After identifying this state, the system will skip the regular prediction output logic of the decision bus, no longer rely on the decision bus to calculate the output result of the prefetch window, and directly use the preset minimum size information as the target size information of the prefetch window. At the same time, it can trigger the setting of the mutation flag, thereby freezing the correction function of the feedback control module and preventing the feedback control module from making incorrect parameter adjustments during periods of abnormal bandwidth.

[0100] The preset minimum size information, also known as the safety lower bound, is used to limit the data scale of prefetching operations in extremely degraded scenarios, ensuring that the high-priority model weight subset necessary for inference computation can be prefetched first. In other words, when the SLC cache is exhausted, because there is no time to make fine-grained decisions, the normal decision-making of the decision bus is skipped, and the window is directly set to the safety lower bound. Only the highest-priority weight subset necessary for the current layer's computation is prefetched. For example, only the Q / K / V (Query, Key, Value) projection weights of the attention layer are prefetched, and the entire weight of the FFN (Feed-Forward Network) layer is not prefetched. Simultaneously, the mutation flag is set, freezing the correction function of the feedback control module, and waiting for the window to recover later.

[0101] In this embodiment, by directly using the preset minimum size information as the target size information of the prefetch window when the SLC cache is exhausted, the scale of prefetched data can be quickly compressed in the scenario where the pseudo SLC cache is full and bandwidth suddenly degrades, so as to prioritize the timely prefetching of the model weights necessary for inference and prevent computational stagnation in large model inference services.

[0102] In one implementation, for the throttling state, the window size remains at the safe lower limit, the decision output by the decision bus is not executed, and the bandwidth recovery signal is continuously monitored, i.e., whether the window has recovered. For the GC stress state, no preventative degradation is performed; instead, a GC fluctuation signal is sent to the feedback control module to increase the integral term damping. Therefore, under GC stress, the prefetch is performed directly using the window size output by the decision bus. In other words, the bandwidth remains within the normal range under GC stress, so there is no need to shrink the window. Increasing the integral term damping is sufficient to prevent the feedback control module from misjudging, and no special window size processing is required. The fluctuations can be smoothed by the damping of the feedback control module alone.

[0103] In step 450, the target monitoring information represents the signal sensing information used to confirm the current state information. The target monitoring information is monitoring data filtered from all signal sensing information; different state information can correspond to different target monitoring information. The target monitoring information is used to continuously observe changes in the operating conditions of the storage device to determine whether the recovery conditions of the prefetch window are met. The target monitoring information can reuse various signal sensing information collected in the previous state identification stage, without requiring a new independent hardware acquisition path. For example, for a throttling state, the target monitoring information includes bandwidth estimation and trend ratio; for an SLC cache exhaustion state, the target monitoring information includes fast line value and SLC cache level information; for a thermal risk warning state, the target monitoring information includes temperature acceleration and trend ratio; and for a GC pressure state, the target monitoring information includes I / O queue monitoring information and bandwidth estimation.

[0104] The target monitoring information is determined based on the current status information of the storage device. This information is monitored in real-time or periodically to determine if it meets the preset window recovery conditions. If so, a window recovery coefficient corresponding to the current status information is determined based on a preset correlation. This window recovery coefficient is then used to adjust the target size information of the prefetched window to obtain the recovered size information of the prefetched window. If not, monitoring of the target monitoring information continues until the preset window recovery conditions are met.

[0105] The preset window recovery conditions will vary depending on the status information of the storage device. That is, different target monitoring information can correspond to different window recovery conditions, so as to implement different recovery strategies based on different degradation reasons.

[0106] For preventative downgrades caused by a thermal risk warning, the target monitoring information can be the temperature smoothing value. The monitoring checks whether the temperature smoothing value begins to decrease (the first-order temperature difference turns negative) and whether it has fallen below a preset trigger threshold. If this condition is met, the thermal risk recovery process begins. If the temperature continues to rise, the preventative window is maintained without recovery, because the failure to reduce the temperature means the thermal throttling risk remains.

[0107] For degradation caused by SLC cache exhaustion, monitor whether the I / O queue depth has returned to normal (reduced to within twice the preset normal value) and whether the bandwidth has recovered to more than 60% of the preset normal range. If both conditions are met, it is determined that the SLC cache has been released by garbage collection, and the recovery process can begin. If the I / O queue depth is still abnormal, it means that garbage collection is still in progress, and recovery will not proceed.

[0108] When recovery is deemed feasible, different window recovery coefficients and recovery strategies can be determined for different status information. This embodiment employs a tiered recovery strategy, performing recovery step by step. The window recovery coefficient used at each level can also be different.

[0109] For the recovery of thermal risk warnings, a slow recovery is adopted, with each level of window recovery having a coefficient of 5%, increasing the prefetch window size from the target size by 5%. After a preset first waiting period, the system is allowed to operate stably under the new window size, collecting temperature and bandwidth information. If the bandwidth estimate stabilizes or increases and the temperature continues to decrease within the preset waiting period, the window size is increased by another 5%. If the bandwidth decreases again or the temperature rebounds, the recovery is paused, maintaining the current recovered window size. The above steps are repeated, increasing by 5% each time, with a preset first waiting period interval, until the window size recovers to the normal output value of the decision bus.

[0110] For recovery from SLC cache exhaustion, this is a fast recovery. The first-level window recovery factor can be 40%, raising the window from the safe lower bound to 40% of the normal window. Wait for a preset second wait period, which can be shorter than the first, as SLC recovery is bursty and does not require long observation. If all prefetchings are completed on time and the hit rate is normal within the second wait period, increase the factor by another 20%, i.e., the second-level window recovery factor is 20%. If prefetching is incomplete, pause the recovery. Repeat the above steps, starting from the second level. The window recovery factor can be increased by the same amount each time, for example, increasing by 20% each time, with an interval of one second wait period, until the window recovers to its normal value.

[0111] For GC pressure recovery, this recovery is a medium-speed recovery. Recovery begins once the I / O queue depth has returned to normal and stabilized within a preset third time period. The window recovery coefficient for each level can be the same, increasing by 10% each time, with a third waiting period between each increase, until the window returns to its normal value. GC pressure states do not need to recover the window from a degraded state because they do not have a degraded window to begin with. GC pressure recovery refers to the gradual release of the integral term damping of the feedback control module as the GC pressure signal returns from fluctuation to stability, restoring it to normal control parameters. This is the recovery of control parameters, not the window size, and is a different dimension of recovery from thermal risk recovery and SLC depletion recovery.

[0112] In other words, the window recovery coefficient controls the upward adjustment range of the prefetch window. Different state information can correspond to different window recovery coefficients to adapt to the recovery characteristics of different causes. For some states, the recovery process can also be configured with an observation and waiting mechanism. After performing a window size adjustment, the target monitoring information is continuously observed. Only if the window recovery conditions are continuously met will the next round of window raising be executed. If the target monitoring information deviates from the window recovery conditions again, the recovery process can be paused to maintain the current window size, avoiding blindly enlarging the prefetch window before the operating conditions have truly improved. The recovered size information of the prefetch window is the window size obtained after adjusting the window recovery coefficient. The recovered size information can be used as the window size for subsequent prefetch operations until the prefetch window returns to the window size corresponding to normal operation.

[0113] In this embodiment, the corresponding target monitoring information is determined based on the status information. When the target monitoring information meets the window recovery conditions, the matching window recovery coefficient is retrieved according to the preset correlation relationship. The target size information of the prefetch window is adjusted to obtain the recovered size information. Adaptive recovery strategies can be adopted for different types of bandwidth degradation conditions. The prefetch window is gradually raised during the improvement of storage device operating conditions, taking into account both the stability of the control loop and the utilization efficiency of storage bandwidth.

[0114] During window size restoration, the feedback control module can be cascaded. During the downgrade phase, the feedback control module is completely frozen and cannot perform feedback corrections. In the initial stage of restoration, the feedback control module allows window size corrections, but the step size is limited to 30% of the normal value. In the middle stage, the correction step size increases to 50% to 60% of the normal value. In the final stage, the correction step size increases to 80% of the normal value. After complete restoration, the step size returns to 100% of the normal value. In this embodiment, the current state (initial, middle, or final stage) can be determined based on the ratio between the window size and the preset normal size during the restoration process. For example, if the current restored size is only 20% of the normal size, it is in the initial stage of restoration.

[0115] Different status information corresponds to different tiered unfreezing strategies. That is, the corresponding tiered unfreezing strategy can be determined based on the current status information, and then the feedback control module can be unfrozen according to the corresponding tiered unfreezing strategy to control the recovery process of the prefetch window. For example, under the hot risk warning state, the correction step size in the initial stage of recovery is 30%, and the correction step size in the final stage is 60%; under the SLC cache exhaustion state, the step size in the early stage of recovery is 30%, in the middle stage it is 50%, and in the final stage it is 83%; under GC pressure state, the entire process is limited to 50%.

[0116] In any state, if the bandwidth decreases again after two consecutive recovery cycles (e.g., the trend ratio turns negative again), the recovery is considered a failure. The system can then revert to the previous stable recovery size information and re-enter the recovery trigger condition detection. If recovery fails more than three times in the same state, recovery will no longer be attempted until the bandwidth environment fundamentally improves, for example, the temperature drops to ten degrees below the lower limit of the safe range, or a significant amount of space is freed up on the storage device.

[0117] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0118] It should be noted that in various specific embodiments of this application, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0119] Description of apparatus and devices in embodiments of this application Reference Figure 5 , Figure 5This is a schematic diagram of the structure of a weight prefetching device 500 for large model inference provided in an embodiment of this application. The weight prefetching device 500 for large model inference includes: The information acquisition module 501 is used to acquire signal sensing information at the current moment; wherein, the signal sensing information includes at least one of temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information. The temperature information represents the controller temperature of the flash memory controller and the packaging temperature of the NAND flash memory chip. The SLC cache level information represents the level of SLC cache usage. The I / O queue monitoring information represents the depth of the NVMe submission queue and completion queue. The bandwidth information represents the changes in bandwidth. The state determination module 502 is used to determine the state information of the storage device at the current moment based on the signal sensing information; wherein, the state information represents the bandwidth degradation condition of the storage device; The window adjustment module 503 is used to determine the target size information of the prefetch window based on the state information of the storage device at the current moment; wherein, the prefetch window is used to control the number of model weights prefetched during large model inference; The weight prefetching module 504 is used to obtain model weights from NAND flash memory chips and store them in a preset cache area based on the target size information of the prefetching window.

[0120] In one example, bandwidth information includes a bandwidth estimate and a trend ratio. The bandwidth estimate represents the estimated storage throughput capacity, and the trend ratio represents the upward or downward trend of the bandwidth. The information acquisition module 501 is specifically used for: Obtain the amount of data and time consumption information of the model weights prefetched in the previous time step; Based on the amount of model weight data and time consumption information pre-fetched in the previous time step, the fast line value and slow line value are determined based on the preset dual-speed exponential weighted moving average smoother. The fast line value represents the bandwidth value of the fast line output of the preset dual-speed exponential weighted moving average smoother, and the slow line value represents the bandwidth value of the slow line output of the preset dual-speed exponential weighted moving average smoother. Based on the fast line value and the slow line value, determine the bandwidth estimate and trend ratio.

[0121] In one example, the information acquisition module 501 is specifically used for: The instantaneous bandwidth value is determined based on the amount of model weight data and the time consumption information prefetched in the previous time step; whereby the instantaneous bandwidth value represents the actual storage throughput capacity. Input the instantaneous bandwidth value into the fast line of the preset dual-speed exponentially weighted moving average smoother to obtain the fast line value, and input the instantaneous bandwidth value into the slow line of the preset dual-speed exponentially weighted moving average smoother to obtain the slow line value.

[0122] In one example, bandwidth information includes a bandwidth estimate and a trend ratio. The bandwidth estimate represents the estimated storage throughput, and the trend ratio represents the rise and fall trend of the bandwidth. The state determination module 502 is specifically used for: If the bandwidth estimate is within a preset bandwidth range and the trend ratio is within a preset trend ratio range, then the storage device's status information at the current moment is determined to be in a throttled state; where the throttled state indicates that the size information of the prefetch window has been adjusted downwards.

[0123] In one example, the bandwidth information includes the fastline value, which represents the bandwidth value of the fastline output of a preset dual-speed exponentially weighted moving average smoother; the state determination module 502 is specifically used for: If the bandwidth estimate is not within the preset bandwidth range, or the trend ratio is not within the preset trend ratio range, then the decrease in the fast line value within the preset time period is determined. If the decrease value is within the preset range and the SLC cache level information exceeds the preset level threshold, then the storage device is determined to be in an SLC cache exhaustion state at the current moment; whereby the SLC cache exhaustion state indicates that the available space of the pseudo-SLC cache is filled.

[0124] In one example, the temperature information includes temperature acceleration; the state determination module 502 is specifically used for: If the decrease is not within the preset range, or the SLC cache level information does not exceed the preset level threshold, then when the temperature acceleration is greater than the preset temperature threshold and the trend ratio is negative, the storage device's status information at the current moment is determined to be a thermal risk warning state; whereby the thermal risk warning state indicates that the bandwidth will decrease due to thermal throttling.

[0125] In one example, the state determination module 502 is specifically used for: If the temperature acceleration is not greater than the preset temperature threshold and the trend ratio is not negative, then when the I / O queue monitoring information meets the first preset condition and the bandwidth estimate meets the second preset condition, the storage device's current state information is determined to be GC pressure state; whereby GC pressure state characterizes the periodic bandwidth jitter caused by garbage collection preempting bandwidth.

[0126] In one example, window resizing module 503 is specifically used for: If the storage device is in a thermal risk warning state at the current moment, the initial adjustment size information of the prefetch window is determined based on the current size information of the prefetch window; wherein, the thermal risk warning state indicates that the bandwidth will decrease due to thermal throttling; the initial adjustment size information indicates the size information after the prefetch window is initially shrunk. Based on the initial size information and the preset window shrinkage coefficient, determine the target size information of the pre-fetched window.

[0127] In one example, window resizing module 503 is specifically used for: If the storage device is in an SLC cache exhaustion state at the current moment, the preset minimum size information will be determined as the target size information of the prefetch window; where SLC cache exhaustion state indicates that the bandwidth will decrease due to hot throttling; the initial size information indicates that the available space of the pseudo SLC cache is filled.

[0128] In one example, it also includes: The target monitoring module is used to determine target monitoring information based on the status information of the storage device at the current moment; wherein, the target monitoring information represents the signal sensing information used to confirm the status information at the current moment; The coefficient determination module is used to determine the window recovery coefficient corresponding to the current state information based on the preset correlation relationship if the target monitoring information meets the preset window recovery conditions; wherein, the preset correlation relationship represents the correlation relationship between the state information and the window recovery coefficient. The window restoration module is used to adjust the target size information of the prefetched window according to the window restoration coefficient corresponding to the current state information, so as to obtain the restored size information of the prefetched window.

[0129] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0130] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0131] Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 600 of this embodiment includes: at least one processor 601 ( Figure 6 (Only one is shown in the diagram), memory 602, and computer program 603 stored in said memory 602 and executable on said at least one processor 601, wherein said processor 601 executes said computer program 603 to implement the steps in any of the above embodiments of the weight prefetching method for large model inference.

[0132] Electronic device 600 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This electronic device may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 600 and does not constitute a limitation on electronic device 600. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0133] The processor 601 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0134] In some embodiments, the memory 602 may be an internal storage unit of the electronic device 600, such as a hard disk or memory of the electronic device 600. In other embodiments, the memory 602 may be an external storage device of the electronic device 600, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 600. Furthermore, the memory 602 may include both internal and external storage units of the electronic device 600. The memory 602 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 602 can also be used to temporarily store data that has been output or will be output.

[0135] This application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0136] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0137] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0138] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0139] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0140] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0141] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A weight prefetching method for large model inference, characterized in that, include: Obtain signal sensing information at the current moment; wherein, the signal sensing information includes at least one of temperature information, SLC cache level information, I / O queue monitoring information, and bandwidth information, wherein the temperature information represents the controller temperature of the flash memory controller and the packaging temperature of the NAND flash memory chip, the SLC cache level information represents the level of SLC cache usage, the I / O queue monitoring information represents the depth of the NVMe submission queue and completion queue, and the bandwidth information represents the bandwidth change; Based on the signal sensing information, the state information of the storage device at the current moment is determined; wherein, the state information represents the bandwidth degradation condition of the storage device; Based on the state information of the storage device at the current moment, the target size information of the prefetch window is determined; wherein, the prefetch window is used to control the number of model weights prefetched during large model inference; Based on the target size information of the prefetch window, model weights are obtained from the NAND flash memory chip and stored in a preset cache area.

2. The method according to claim 1, characterized in that, The bandwidth information includes a bandwidth estimate and a trend ratio. The bandwidth estimate represents the estimated storage throughput capacity, and the trend ratio represents the upward and downward trend of the bandwidth. The acquisition of signal sensing information at the current moment includes: Obtain the amount of data and time consumption information of the model weights prefetched in the previous time step; Based on the amount of model weight data and time consumption information pre-fetched at the previous time step, the fast line value and slow line value are determined based on the preset dual-speed exponential weighted moving average smoother; wherein, the fast line value represents the bandwidth value of the fast line output of the preset dual-speed exponential weighted moving average smoother, and the slow line value represents the bandwidth value of the slow line output of the preset dual-speed exponential weighted moving average smoother. The bandwidth estimate and the trend ratio are determined based on the fast line value and the slow line value.

3. The method according to claim 2, characterized in that, The step of determining the fast line value and slow line value based on the amount of model weight data and time consumption information pre-fetched from the previous time step, using a preset dual-speed exponential weighted moving average smoother, includes: The instantaneous bandwidth value is determined based on the amount of model weight data and the time consumption information prefetched in the previous time step; wherein, the instantaneous bandwidth value represents the actual storage throughput capacity. The instantaneous bandwidth value is input to the fast line of the preset dual-speed exponentially weighted moving average smoother to obtain the fast line value, and the instantaneous bandwidth value is input to the slow line of the preset dual-speed exponentially weighted moving average smoother to obtain the slow line value.

4. The method according to claim 1, characterized in that, The bandwidth information includes a bandwidth estimate and a trend ratio. The bandwidth estimate represents the estimated storage throughput capacity, and the trend ratio represents the upward and downward trend of the bandwidth. Determining the state information of the storage device at the current moment based on the signal sensing information includes: If the bandwidth estimate is within a preset bandwidth range and the trend ratio is within a preset trend ratio range, then the storage device is determined to be in a throttling state at the current moment; wherein, the throttling state indicates that the size information of the prefetch window has been adjusted downwards.

5. The method according to claim 4, characterized in that, The bandwidth information includes the fast line value, which represents the bandwidth value of the fast line output of the preset dual-speed exponentially weighted moving average smoother. Determining the state information of the storage device at the current moment based on the signal sensing information includes: If the estimated bandwidth value is not within the preset bandwidth range, or the trend ratio is not within the preset trend ratio range, then the decrease magnitude of the fast line value within the preset time period is determined. If the decrease value is within a preset range and the SLC cache level information exceeds a preset level threshold, then the storage device is determined to be in an SLC cache exhaustion state at the current moment; wherein, the SLC cache exhaustion state indicates that the available space of the pseudo-SLC cache is full.

6. The method according to claim 5, characterized in that, The temperature information includes temperature acceleration; Determining the state information of the storage device at the current moment based on the signal sensing information includes: If the decrease in amplitude is not within the preset amplitude range, or the SLC cache level information does not exceed the preset level threshold, then when the temperature acceleration is greater than the preset temperature threshold and the trend ratio is negative, the storage device is determined to be in a thermal risk warning state at the current moment; wherein, the thermal risk warning state indicates that the bandwidth will decrease due to thermal throttling.

7. The method according to claim 6, characterized in that, Determining the state information of the storage device at the current moment based on the signal sensing information includes: If the temperature acceleration is not greater than a preset temperature threshold and the trend ratio is not negative, then when the I / O queue monitoring information meets the first preset condition and the bandwidth estimate meets the second preset condition, the state information of the storage device at the current moment is determined to be a GC pressure state; wherein, the GC pressure state characterizes the periodic bandwidth jitter caused by garbage collection preempting bandwidth.

8. The method according to claim 1, characterized in that, Determining the target size information of the prefetch window based on the current state information of the storage device includes: If the storage device is in a thermal risk warning state at the current moment, the initial adjustment size information of the prefetch window is determined based on the current size information of the prefetch window; wherein, the thermal risk warning state indicates that the bandwidth will decrease due to thermal throttling; the initial adjustment size information indicates the size information after the prefetch window is initially shrunk. Based on the initial size information and the preset window shrinkage coefficient, determine the target size information of the pre-fetched window; or, If the storage device is in an SLC cache exhaustion state at the current moment, then the preset minimum size information is determined as the target size information of the prefetch window; wherein, the SLC cache exhaustion state indicates that the bandwidth will decrease due to hot throttling; the initial size information indicates that the available space of the pseudo SLC cache is filled.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Based on the status information of the storage device at the current moment, target monitoring information is determined; wherein, the target monitoring information represents signal sensing information used to confirm the status information at the current moment; If the target monitoring information meets the preset window recovery conditions, then the window recovery coefficient corresponding to the state information at the current moment is determined according to the preset correlation relationship; wherein, the preset correlation relationship represents the correlation relationship between the state information and the window recovery coefficient; Based on the window recovery coefficient corresponding to the state information at the current moment, the target size information of the prefetched window is adjusted to obtain the recovered size information of the prefetched window.

10. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-9.

Images