Storage hierarchical acceleration system for large-model high-concurrency reasoning

Abstract

The invention relates to the technical field of artificial intelligence infrastructure, in particular to a large-model high-concurrency reasoning storage hierarchical acceleration system, which comprises an access popularity acquisition module, a pressure analysis module, a migration execution module and a heterogeneous storage pool, the access popularity acquisition module is used for acquiring the access frequency A and the access delay D of model parameters in real time. The effects of sensing system pressure in real time and accurately triggering migration are achieved by arranging the access popularity acquisition module and the pressure analysis module, the access popularity acquisition module continuously monitors the access frequency and delay of model parameters, and the pressure analysis module calculates and stores a pressure index based on a historical peak value and a dynamic threshold value. When the index exceeds the preset threshold value, migration operation is triggered immediately, the problem that delay exceeds the standard due to storage I / O bottleneck in the financial transaction peak period is solved, the system can still keep millisecond-level response under tens of thousands of concurrent requests per second, and risk misinformation and missing report caused by delay jitter are avoided.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence infrastructure technology, and more specifically, to a storage tiered acceleration system for high-concurrency inference of large models. Background Technology

[0002] Large-scale language models (LLMs), such as family models, are being increasingly integrated into various real-time online services due to their powerful generation and understanding capabilities. However, the performance and stability of the large-model inference engines supporting these services face severe challenges when dealing with sudden surges of extremely high concurrency requests. A typical bottleneck lies in storage I / O. Existing inference systems typically use a single, high-performance storage medium (such as an all-NVMe SSD array) to store model parameters in pursuit of low latency. This approach performs well under small-scale or steady-state concurrency, but problems become apparent when faced with sudden, extremely high surges in inference requests, especially when applied to real-time transaction risk monitoring and fraud analysis systems in large fintech companies.

[0003] The core of a large fintech company's real-time transaction risk monitoring and fraud analysis system is to analyze massive transaction flows (e.g., tens of thousands of transactions per second) in real time within a millisecond-level time window after a transaction occurs, using a model of equal size to identify potential fraud patterns, abnormal behavior, and compliance risks. This is a scenario with extremely high requirements for ultra-low latency and ultra-high throughput. The system's pressure peaks before and after the opening and closing of major global financial markets and during the release of important economic data (e.g., the overlapping opening sessions of exchanges in New York, London, Tokyo, and Hong Kong). During these periods, global trading volume surges, and the system needs to handle an explosive increase in concurrent inference requests in a very short time. In this scenario, the response time of model inference (P99 latency) must be strictly controlled within milliseconds; any latency fluctuations may lead to missed or false alarms, resulting in direct economic losses or regulatory penalties. At the same time, the models themselves are usually very large (tens of gigabytes or even hundreds of gigabytes). How to efficiently and economically meet the massive, random access requirements for model parameters during these peak periods while ensuring extremely low access latency is a pain point that existing single storage architectures cannot solve.

[0004] Existing technologies, in such demanding financial-grade environments with high concurrency and low latency, either fail to deploy full-scale high-speed models due to excessively high storage costs, or suffer from excessive inference latency due to insufficient storage performance, severely limiting the practical application value of large models in this critical business area. Therefore, an innovative storage management solution is urgently needed that can dynamically tier storage based on the access frequency of model parameters. This solution can ensure ultra-fast access to core, frequently accessed data while utilizing lower-cost storage media to handle less frequently accessed data, thereby achieving an optimal balance between performance and cost and significantly improving the system's resilience and stability in handling extremely high concurrency peaks. Summary of the Invention

[0005] The purpose of this invention is to provide a storage-tiered acceleration system for high-concurrency inference of large models, in order to solve the problem mentioned in the background art that the existing technology is either unable to deploy the full high-speed model due to excessive storage costs or causes excessive inference latency due to insufficient storage performance in the harsh environment of financial-grade high concurrency and low latency, which seriously restricts the application value of large models in this key business area.

[0006] To achieve the above objectives, the present invention aims to provide a storage tiered acceleration system for high-concurrency inference of large models, the system comprising an access heat acquisition module, a stress analysis module, a migration execution module, and a heterogeneous storage pool; The access frequency acquisition module is used to collect the access frequency and access latency of the model parameters in real time; The pressure analysis module is used to calculate the storage pressure index based on the data collected by the access popularity acquisition module, and, in conjunction with a preset threshold, determine whether a data migration operation needs to be triggered. Specifically: The storage pressure index is calculated based on access frequency, access latency, historical peak access frequency, and preset weighting coefficients. A preset storage pressure index threshold is set, and the calculated storage pressure index is compared with the preset storage pressure index threshold. When the storage pressure index exceeds the threshold, a data migration operation instruction is triggered to the migration execution module. The heterogeneous storage pool is used to integrate storage media with different performance and cost to form a hierarchical storage structure, which includes a high-speed storage layer, a medium-speed storage layer and a cold storage layer. The high-speed storage layer is used to store frequently accessed hot data, the medium-speed storage layer is used to store data with medium access frequency, and the cold storage layer is used to store data with low access frequency. The migration execution module is used to perform data migration operations according to the data migration operation instructions sent by the stress analysis module, and migrate high-frequency parameters to the high-speed storage layer.

[0007] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0008] 1. This invention achieves real-time perception of system pressure and precise triggering of migration by setting up an access heat acquisition module and a pressure analysis module. The access heat acquisition module continuously monitors the access frequency and latency of model parameters, while the pressure analysis module calculates the storage pressure index based on historical peak values ​​and dynamic thresholds. When the index exceeds the preset threshold, the migration operation is triggered immediately. This solves the problem of excessive latency caused by storage I / O bottlenecks during peak financial transaction periods, enabling the system to maintain millisecond-level response even with tens of thousands of concurrent requests per second, and avoiding false alarms and missed alarms caused by latency jitter.

[0009] 2. This invention achieves intelligent optimization of high-speed storage resource utilization by setting a priority queue mechanism for the migration execution module. The migration execution module calculates the heat decay factor and migration weight index for each parameter, generates a global priority queue, and migrates only the top N% of key parameters to the high-speed storage layer. This breaks through the resource waste bottleneck of traditional full migration, reduces the cost of high-speed storage while ensuring fast access to core hot data, and perfectly solves the pain point of excessively high cost of full high-speed storage deployment.

[0010] 3. This invention achieves an adaptive effect in responding flexibly to financial peaks by setting up a three-layer pressure classification and a burst traffic correction mechanism. Based on the storage pressure index, the system status is divided into three levels: basic, medium and severe overload, and the migration ratio is dynamically adjusted. When the concurrent growth rate is detected to exceed 100%, an emergency increment is automatically added, thereby overcoming the problem of service crashes caused by the surge of requests during the overlapping period of exchange opening.

[0011] 4. This invention achieves the effect of hierarchical protection of critical business continuity by setting a three-level response strategy for the concurrent scheduling module. The first-level response dynamically allocates bandwidth according to transaction type to ensure the smooth operation of high-priority services such as fraud analysis. The second-level response activates backup high-speed nodes to avoid single points of failure. The third-level response preloads predicted hotspots into memory and degrades cold data to HDD, thereby ensuring the availability of the system under extreme pressure and solving the fatal defect of service interruption caused by sudden traffic. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the module interaction of the storage tiered acceleration system for large-model high-concurrency inference of the present invention. Detailed Implementation

[0013] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0014] In one specific embodiment, a large fintech company's real-time transaction risk monitoring and fraud analysis system utilizes large models to perform real-time analysis of massive transaction flows (e.g., tens of thousands of transactions per second) to identify potential fraud patterns, abnormal behaviors, and compliance risks. Figure 1 As shown, the storage tiered acceleration system for high-concurrency inference of large models includes an access heat acquisition module, a stress analysis module, a migration execution module, a heterogeneous storage pool, a concurrent scheduling module, a data prefetching module, and a time-sharing scheduling engine. The functions of each module are as follows: The access frequency acquisition module is used to collect the access frequency A and access latency D of the model parameters in real time.

[0015] The stress analysis module is used to calculate the storage stress index P based on the data collected by the access popularity acquisition module, and combine it with a preset storage stress index threshold P. th Determine whether a data migration operation needs to be triggered.

[0016] The heterogeneous storage pool integrates a high-speed storage layer, a medium-speed storage layer, and a cold storage layer. In the real-time transaction risk monitoring and fraud analysis system, frequently accessed hot data (such as real-time transaction data and fraud pattern identification parameters) is stored in the high-speed storage layer to ensure extremely low access latency. Data with moderate access frequency (such as historical transaction data) is stored in the medium-speed storage layer, while data with low access frequency (such as long-term archived transaction records) is stored in the cold storage layer. This tiered storage structure effectively balances performance and cost, improving the overall efficiency of the system.

[0017] The migration execution module is used to perform data migration operations according to the data migration operation instructions sent by the stress analysis module, and to migrate high-frequency parameters to the high-speed storage layer.

[0018] The concurrent scheduling module is used when the storage pressure index P > 0.8P. th Resource allocation is performed in a timely manner.

[0019] The data prefetching module is used to predict the access sequence of hot parameters within the next 5-10 seconds based on an LSTM neural network. When the predicted access probability is greater than 90%, a preloading operation is automatically triggered to migrate the relevant parameters from the cold storage layer to the high-speed storage layer in advance. By predicting hot parameters and preloading them in advance, access latency is further reduced, and the system's response speed and processing efficiency are improved.

[0020] The time-sharing scheduling engine has a built-in global financial exchange calendar to track the opening or closing times of various exchanges in real time and automatically perform the following operations during overlapping opening periods: Set the preset storage pressure index threshold P th Reduced by 20%; Expand the capacity of the high-speed storage layer by 300%; Increase the compression ratio of cold data to 8:1.

[0021] By automatically adjusting system parameters, the system effectively copes with storage pressure during peak periods such as overlapping exchange opening times, improves system elasticity and stability, and ensures normal operation of the system under high concurrency requests.

[0022] In a real-time transaction risk monitoring and fraud analysis example: The first step involves continuously monitoring the access frequency (A) and access latency (D) of large model parameters using an access frequency acquisition module. When a specific model parameter (such as a threshold parameter used to identify fraudulent transactions) is detected to be frequently accessed within a very short period, this module records this access frequency information in real time. By monitoring the access frequency of model parameters in real time, an accurate data foundation is provided for subsequent stress analysis and data migration, ensuring that the system can respond promptly to changes in storage pressure.

[0023] The second step involves calculating the storage pressure index P during peak system trading periods (such as when the opening hours of major global financial markets overlap). This occurs when access frequency (A) and access latency (D) increase significantly. At this time, the stress analysis module uses data provided by the access intensity collection module to calculate the storage pressure index P. Specifically: According to the formula Calculate the storage pressure index P, where, and Preset weighting coefficients are used to adjust the relative importance of access frequency and latency in the storage stress index calculation, to meet the following requirements. , This represents the historical peak access frequency, used to normalize the current access frequency. A preset latency threshold is used to normalize the current latency, dynamically calculated based on historical latency data. ,in The formula takes into account both access frequency and latency as historical peak latency, and is used to evaluate the storage pressure of the system in real time. Preset storage pressure index threshold P th The calculated storage pressure index P is then compared with the preset storage pressure index threshold P. th For comparison, when P > P th When the system pressure exceeds a preset threshold, a data migration operation command is triggered and sent to the migration execution module. By calculating the storage pressure index, the system can sense the system pressure in real time. When the pressure exceeds the preset threshold, the migration operation is triggered precisely, effectively avoiding latency issues caused by storage I / O bottlenecks and ensuring that the system can maintain millisecond-level response even under high concurrency requests.

[0024] Among them, the preset weight coefficient and The calculation method is as follows: Collect current concurrency and historical peak concurrency; According to the formula Calculate the preset weighting coefficients ; According to the formula Calculate the preset weighting coefficients . use and The adjustments ensure that the system can reasonably sense storage pressure based on changes in access frequency and latency under different load scenarios, thereby triggering data migration operations in a timely manner and avoiding excessive latency.

[0025] Thirdly, after receiving the data migration operation instruction, the migration execution module intelligently selects the model parameters to be migrated. It calculates the heat decay factor and migration weight index for each parameter, generates a priority queue, and migrates only frequently accessed hot parameters (such as model parameters used for high-frequency trading analysis) to a high-speed storage layer (such as NVMe SSD). This process ensures that hot data can be accessed quickly under high concurrency requests, thereby reducing latency. Specifically: According to the formula Calculate the heat decay factor H of this parameter, where, is the base of the natural logarithm, k is the decay coefficient used to control the rate of heat decay, and t is the current time. The last access time represents the time when the model parameter was last accessed. The formula is used to calculate the heat decay factor H of the model parameter, which is the degree to which the heat of the parameter decreases over time. By introducing the decay coefficient k, the system can adjust the rate of heat decay according to actual needs, ensuring that the heat calculation can accurately reflect the real-time access status of the parameter and provide a reliable basis for data migration. Based on the heat decay factor H, and combined with the formula Calculate the migration weight index M for this parameter; The calculated migration weight index M is sorted in descending order to generate a priority queue, ensuring that frequently accessed hot parameters can be migrated to the high-speed storage layer first, thereby optimizing system performance. Only the top N% of model parameters in the queue are migrated to the high-speed storage layer, where N is the migration ratio, and this is dynamically adjusted and updated in real time. By intelligently selecting the model parameters to be migrated and generating a priority queue, only frequently accessed hot parameters are migrated to the high-speed storage layer, optimizing the utilization of high-speed storage resources, reducing migration costs, and ensuring extremely fast access to core hot data.

[0026] The dynamic adjustment method for the migration ratio N is as follows: Monitor real-time storage pressure index: Obtain the current storage pressure index P calculated by the pressure analysis module, and read the preset pressure threshold level: , , ; Hierarchical matching migration ratio: If Then set N=15, if Then set N=30, if Then set N=50; Adaptive adjustment for burst traffic: Calculate the growth rate of concurrent requests And monitor it; when the growth rate of concurrent requests is detected... When the migration rate is >100% / second, an emergency increment is added to the current migration rate N value. And ensure the final migration ratio .

[0027] Among them, the growth rate of concurrent requests The calculation method is as follows: Define the time window and sampling points: Set the monitoring time window =1 second, mark consecutive sampling points on the time axis: current time 1 second ago 2 seconds ago ; Obtain concurrent request volume time-series data: Read the request volume recorded by the concurrent scheduling module: Request volume per moment , Request volume per moment , Request volume per moment ; According to the formula Calculate the instantaneous growth rate ; According to the formula Calculate the smoothed growth rate ; According to the formula Output the final concurrent request growth rate. By introducing a smoothing growth rate The system can reduce the impact of instantaneous fluctuations on the calculation results, thereby more accurately judging the growth trend of the request volume.

[0028] Through the collaborative work of the above modules, the system can efficiently process massive transaction flows in the real-time transaction risk monitoring and fraud analysis system of large financial technology companies, achieving real-time analysis with ultra-low latency and ultra-high throughput, thereby effectively identifying potential fraud patterns, abnormal behaviors and compliance risks.

[0029] In summary, by setting up access frequency acquisition and stress analysis modules, the system can perceive system stress in real time and accurately trigger migration. The access frequency acquisition module continuously monitors the access frequency and latency of model parameters, while the stress analysis module calculates the storage stress index based on historical peak values ​​and dynamic thresholds. When the index exceeds the preset threshold, the migration operation is triggered immediately. This solves the problem of excessive latency caused by storage I / O bottlenecks during peak financial transaction periods, enabling the system to maintain millisecond-level response even with tens of thousands of concurrent requests per second, and avoiding false alarms and missed alarms caused by latency jitter.

[0030] Meanwhile, when the storage pressure index P > 0.8P th At this time, the system performs resource allocation through the concurrent scheduling module, specifically: Real-time acquisition of concurrency of the i-th type of request and storage layer response speed ; According to the formula Calculate the load balancing index L, where S max This represents the maximum response speed across all storage layers. The formula is used to evaluate the system's load balancing performance when faced with different types of requests. By comparing the storage layer response speed for a specific request with the fastest response speed, the system can determine whether resource allocation needs to be adjusted to optimize overall performance and response speed. When the load balancing index exceeds a preset threshold, the system will trigger corresponding response strategies, such as allocating more bandwidth or activating standby nodes. Preset load balancing index threshold L th The calculated load balancing index L is then compared with the preset load balancing index threshold L. th For comparison, when L > L th At that time, the concurrent scheduling module executes a three-level response strategy.

[0031] Among them, the storage layer response speed of the i-th type of request The calculation method is as follows: Count the number of times the i-th type of request is hit in each storage layer; According to the formula , as well as Calculate the weights w1 for the high-speed storage layer, w2 for the medium-speed storage layer, and w3 for the cold storage layer, respectively. For the i-th type of request, record its latency data at each storage layer, including the high-speed storage layer access latency. Medium-speed storage layer access latency and cold storage layer access latency ; According to the formula Calculate the storage layer response speed for the i-th type of request. .

[0032] The three-level response strategy is as follows: when When this occurs, a Level 1 response is triggered, according to the formula. Calculate the bandwidth allocated to the i-th type of request. and in accordance with Allocate bandwidth, of which, This represents the total bandwidth of the high-speed layer. when When this occurs, a secondary response is triggered, activating the backup high-speed storage node; when When a three-tiered response is triggered, predicted hotspot parameters are preloaded into memory, and cold data is degraded to the HDD layer. This tiered response strategy effectively ensures the continuity of critical services. The first-tier response dynamically allocates bandwidth to ensure high-priority services operate smoothly; the second-tier response activates backup high-speed nodes to avoid single points of failure; and the third-tier response preloads predicted hotspots into memory and degrades cold data to the HDD, improving the overall availability and stability of the system.

[0033] In summary, this invention, through the collaborative work of its various modules, achieves multiple benefits, including real-time perception of system pressure, precise triggering of migration, optimized utilization of storage resources, and ensuring the continuity of critical business operations. It provides strong support for the real-time transaction risk monitoring and fraud analysis systems of large financial technology companies.

[0034] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A storage-tiered acceleration system for high-concurrency inference of large models, characterized in that: The system includes an access heat acquisition module, a stress analysis module, a migration execution module, and a heterogeneous storage pool; The access frequency acquisition module is used to collect the access frequency and access latency of the model parameters in real time; The pressure analysis module is used to calculate the storage pressure index based on the data collected by the access popularity acquisition module, and, in conjunction with a preset threshold, determine whether a data migration operation needs to be triggered. Specifically: The storage pressure index is calculated based on access frequency, access latency, historical peak access frequency, and preset weighting coefficients. A preset storage pressure index threshold is set, and the calculated storage pressure index is compared with the preset storage pressure index threshold. When the storage pressure index exceeds the threshold, a data migration operation instruction is triggered to the migration execution module. The heterogeneous storage pool is used to integrate storage media with different performance and cost to form a hierarchical storage structure, which includes a high-speed storage layer, a medium-speed storage layer and a cold storage layer. The high-speed storage layer is used to store frequently accessed hot data, the medium-speed storage layer is used to store data with medium access frequency, and the cold storage layer is used to store data with low access frequency. The migration execution module is used to perform data migration operations according to the data migration operation instructions sent by the stress analysis module, and migrate high-frequency parameters to the high-speed storage layer.

2. The storage-layered acceleration system for high-concurrency inference of large models according to claim 1, characterized in that, When performing data migration operations, the migration execution module performs the following operations for each individual model parameter: The heat decay factor of this parameter is calculated based on the last access time and the current time. The migration weight index of this parameter is calculated based on the heat decay factor and the current access frequency; Sort the calculated migration weight indices in descending order to generate a priority queue; Only a certain percentage of the model parameters in the queue are migrated to the high-speed storage layer. This percentage is the migration ratio, which is dynamically adjusted and updated in real time.

3. The storage-layered acceleration system for high-concurrency inference of large models according to claim 2, characterized in that, The method for dynamically adjusting the migration ratio is as follows: Monitor real-time storage pressure index and read preset multi-level pressure threshold levels; Based on the current threshold level of the storage pressure index, match and set the corresponding migration ratio; The system detects the growth rate of concurrent requests. When the growth rate exceeds a set threshold, it adds an emergency increment to the current migration ratio and ensures that the final migration ratio does not exceed the set upper limit.

4. The storage-layered acceleration system for high-concurrency inference of large models according to claim 3, characterized in that, The method for calculating the concurrent request growth rate is as follows: Set the monitoring time window and continuous sampling points; Obtain the number of concurrent requests at each sampling time; Calculate the instantaneous growth rate and the smoothed growth rate, and output the final concurrent request growth rate.

5. The storage-layered acceleration system for high-concurrency inference of large models according to claim 1, characterized in that, The system also includes a concurrent scheduling module, used to perform resource allocation when the storage pressure index exceeds a set threshold, specifically: Real-time acquisition of concurrency levels and storage layer response speed for a specific type of request; Calculate the load balancing index based on concurrency and storage layer response speed; A preset load balancing index threshold is set, and the calculated load balancing index is compared with the preset threshold. When the threshold is exceeded, the concurrent scheduling module executes a three-level response strategy.

6. The storage-tiered acceleration system for high-concurrency inference of large models according to claim 5, characterized in that, The method for calculating the storage layer response speed of a certain type of request is as follows: Count the number of times this type of request hits in each storage layer and calculate the weight of each layer; Record the latency data of this type of request in each storage layer; Based on the weights and latency data of each layer, the storage layer response speed for this type of request is calculated.

7. The storage-tiered acceleration system for high-concurrency inference of large models according to claim 5, characterized in that, The three-level response strategy is as follows: When the load balancing index is in the first range, a level 1 response is triggered, and bandwidth is dynamically allocated in a specific manner. When the load balancing index is in the second range, a level 2 response is triggered, activating the backup high-speed storage node. When the load balancing index is in the third range, a level 3 response is triggered, which performs preloading of predicted hot spot parameters and degradation of cold data.

8. The storage-layered acceleration system for high-concurrency inference of large models according to claim 1, characterized in that, The calculation method for the preset weighting coefficient is as follows: Collect current concurrency and historical peak concurrency; Based on the relationship between the current concurrency and the historical peak concurrency, the preset weighting coefficient is calculated.

9. The storage-layered acceleration system for high-concurrency inference of large models according to claim 1, characterized in that, The system also includes a data prefetching module, which is used to predict the hot parameter access sequence in the future based on the LSTM neural network. When the predicted access probability exceeds a set threshold, the preloading operation is automatically triggered to migrate the relevant parameters from the cold storage layer to the high-speed storage layer in advance.

10. The storage-layered acceleration system for high-concurrency inference of large models according to claim 1, characterized in that, The system also includes a time-sharing scheduling engine, which has a built-in global financial exchange calendar to track the opening or closing times of each exchange in real time and automatically perform the following operations during overlapping opening periods: Lower the preset storage pressure index threshold; Expand the capacity of the high-speed storage layer; Improve the compression ratio of cold data.

Images