Storage hierarchical acceleration system for large model high concurrency inference

Abstract

The application relates to the technical field of artificial intelligence infrastructure, in particular to a storage layering acceleration system for high-concurrency reasoning of a large model, which comprises an access frequency acquisition module, a stress analysis module, a migration execution module and a heterogeneous storage pool; the access frequency acquisition module is used for collecting the access frequency A and the access delay D of model parameters in real time. The application achieves the effects of real-time sensing of system stress and accurate triggering of migration by setting the access frequency acquisition module and the stress analysis module; the access frequency acquisition module continuously monitors the access frequency and the delay of model parameters; the stress analysis module calculates a storage stress index based on historical peak values and dynamic threshold values; when the index exceeds a preset threshold value, a migration operation is immediately triggered; the application solves the problem of excessive delay caused by storage I / O bottlenecks during a financial transaction peak period, enables the system to still maintain a millisecond-level response under tens of thousands of concurrent requests per second, and avoids risk false positives and omissions caused by delay jitter.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence infrastructure, in particular to a storage layering acceleration system for large model high concurrency inference. BACKGROUND

[0002] Large language models (LLMs), such as the GPT series, are being increasingly integrated into various real-time online services due to their powerful generation and understanding capabilities. However, the performance and stability of large model inference engines supporting these services face severe challenges when confronted with instantaneous high concurrency requests. A typical bottleneck lies in storage I / O. Existing inference systems usually use a single, high-performance storage medium (such as a full NVMe SSD array) to store model parameters in order to pursue low latency. This solution performs well under small-scale or steady-state concurrency, but when faced with sudden, extremely high peak inference request floods, problems become apparent, especially when applied to real-time transaction risk monitoring and fraud analysis systems in large financial technology companies.

[0003] The core of real-time transaction risk monitoring and fraud analysis systems in large financial technology companies is to use large models to perform real-time analysis on massive transaction streams (such as tens of thousands of transactions per second) within a millisecond time window of transaction occurrence, identifying potential fraud patterns, abnormal behavior, and compliance risks. This is a scenario with extremely demanding requirements for ultra-low latency and ultra-high throughput. The system's pressure peaks around the opening and closing of major global financial markets (e.g., overlapping periods of opening in New York, London, Tokyo, and Hong Kong exchanges) and during the release of major economic data. During these periods, global transaction volume surges, and the system must handle the explosive growth of concurrent inference requests in a very short time. In this scenario, the response time (P99 latency) of model inference must be strictly controlled within milliseconds, and any latency jitter could lead to risk underreporting or misreporting, causing direct economic losses or regulatory penalties. At the same time, models themselves are usually very large (tens of GB or even hundreds of GB), and how to efficiently and economically meet the massive, random access requirements for model parameters during peak periods while ensuring extremely low access latency is a pain point that existing single storage architectures cannot solve.

[0004] Existing technologies in such a financially rigorous high-concurrency, low-latency environment either cannot deploy full-speed models due to high storage costs or cause inference delays to exceed standards due to insufficient storage performance, severely restricting the application value of large models in this critical business field. Therefore, there is an urgent need for an innovative storage management solution that can dynamically layer according to the access hotness of model parameters, ensuring extremely fast access to core hotspot data while using lower-cost storage media to carry data with lower access frequency, thereby achieving the best balance between performance and cost and significantly improving the system's resilience and stability in response to high concurrency peaks. SUMMARY

[0005] The application aims to provide a storage hierarchical acceleration system for large model high concurrency reasoning, to solve the problem that the prior art in the background art cannot deploy full-amount high-speed models due to high storage cost, or causes reasoning delay to exceed the standard due to insufficient storage performance, which seriously restricts the application value of large models in this key business field.

[0006] To achieve the above-mentioned purpose, the application aims to provide a storage hierarchical acceleration system for large model high concurrency reasoning, which comprises an access heat collection module, a stress analysis module, a migration execution module and a heterogeneous storage pool.

[0007] The access heat collection module is used to collect the access frequency and access delay of model parameters in real time.

[0008] The stress analysis module is used to calculate the storage stress index according to the data collected by the access heat collection module, and to judge whether the data migration operation needs to be triggered in combination with the preset threshold value, specifically:

[0009] The storage stress index is calculated based on the access frequency, access delay, historical access frequency peak value and preset weight coefficient.

[0010] The preset storage stress index threshold value is compared with the calculated storage stress index, and when the storage stress index exceeds the threshold value, the data migration operation instruction is triggered to the migration execution module.

[0011] The heterogeneous storage pool is used to integrate storage media with different performance and cost to form a hierarchical storage structure, which comprises a high-speed storage layer, a medium-speed storage layer and a cold storage layer, the high-speed storage layer is used to store hot data with high access frequency, the medium-speed storage layer is used to store data with medium access frequency, and the cold storage layer is used to store data with low access frequency.

[0012] The migration execution module is used to execute the data migration operation according to the data migration operation instruction sent by the stress analysis module, and to migrate the high-frequency parameters to the high-speed storage layer.

[0013] Compared with the prior art, the application has the following advantages:

[0014] 1. This invention achieves real-time perception of system pressure and precise triggering of migration by setting up an access heat acquisition module and a pressure analysis module. The access heat acquisition module continuously monitors the access frequency and latency of model parameters, while the pressure analysis module calculates the storage pressure index based on historical peak values ​​and dynamic thresholds. When the index exceeds the preset threshold, the migration operation is triggered immediately. This solves the problem of excessive latency caused by storage I / O bottlenecks during peak financial transaction periods, enabling the system to maintain millisecond-level response even with tens of thousands of concurrent requests per second, and avoiding false alarms and missed alarms caused by latency jitter.

[0015] 2. This invention achieves intelligent optimization of high-speed storage resource utilization by setting a priority queue mechanism for the migration execution module. The migration execution module calculates the heat decay factor and migration weight index for each parameter, generates a global priority queue, and migrates only the top N% of key parameters to the high-speed storage layer. This breaks through the resource waste bottleneck of traditional full migration, reduces the cost of high-speed storage while ensuring fast access to core hot data, and perfectly solves the pain point of excessively high cost of full high-speed storage deployment.

[0016] 3. This invention achieves an adaptive effect in responding flexibly to financial peaks by setting up a three-layer pressure classification and a burst traffic correction mechanism. Based on the storage pressure index, the system status is divided into three levels: basic, medium and severe overload, and the migration ratio is dynamically adjusted. When the concurrent growth rate is detected to exceed 100%, an emergency increment is automatically added, thereby overcoming the problem of service crashes caused by the surge of requests during the overlapping period of exchange opening.

[0017] 4. This invention achieves the effect of hierarchical protection of critical business continuity by setting a three-level response strategy for the concurrent scheduling module. The first-level response dynamically allocates bandwidth according to transaction type to ensure the smooth operation of high-priority services such as fraud analysis. The second-level response activates backup high-speed nodes to avoid single points of failure. The third-level response preloads predicted hotspots into memory and degrades cold data to HDD, thereby ensuring the availability of the system under extreme pressure and solving the fatal defect of service interruption caused by sudden traffic. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the module interaction of the storage tiered acceleration system for large-model high-concurrency inference of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In a specific embodiment, in a real-time transaction risk monitoring and fraud analysis system of a large financial technology company, a large model is used to perform real-time analysis on a large number of transaction streams (such as tens of thousands per second) to identify potential fraud patterns, abnormal behaviors, and compliance risks, such as Figure 1 As shown in the storage layering acceleration system for high-concurrency inference of large models, it includes an access heat collection module, a stress analysis module, a migration execution module, a heterogeneous storage pool, a concurrent scheduling module, a data prefetching module, and a time-sharing scheduling engine. The functions of each module are as follows:

[0021] The access heat collection module is used to collect the access frequency A and access delay D of model parameters in real time.

[0022] The stress analysis module is used to calculate the storage stress index P according to the data collected by the access heat collection module, and in combination with the preset storage stress index threshold P th , to determine whether to trigger a data migration operation.

[0023] The heterogeneous storage pool integrates a high-speed storage layer, a medium-speed storage layer, and a cold storage layer. In the real-time transaction risk monitoring and fraud analysis system, hot data with high-frequency access (such as real-time transaction data and fraud pattern recognition parameters) is stored in the high-speed storage layer to ensure extremely low access delay. Data with medium-frequency access (such as historical transaction data) is stored in the medium-speed storage layer, and data with low-frequency access (such as long-term archived transaction records) is stored in the cold storage layer. This layered storage structure effectively balances performance and cost, improving the overall efficiency of the system.

[0024] The migration execution module is used to perform data migration operations according to the data migration operation instructions sent by the stress analysis module, and to migrate high-frequency parameters to the high-speed storage layer.

[0025] The concurrent scheduling module is used to perform resource allocation when the storage stress index P is greater than 0.8P th .

[0026] The data prefetching module is used to predict the hot parameter access sequence in the next 5-10 seconds based on the LSTM neural network. When the predicted access probability is greater than 90%, it automatically triggers a preloading operation to migrate the relevant parameters from the cold storage layer to the high-speed storage layer in advance. By predicting hot parameters and preloading in advance, access delay is further reduced, and the response speed and processing efficiency of the system are improved.

[0027] The time-sharing scheduling engine has a built-in global financial exchange calendar to track the opening or closing time of each exchange in real time, and automatically performs the following operations during the overlapping period of the opening of the exchanges:

[0028] The preset storage stress index threshold P th is lowered by 20%.

[0029] The capacity of the high-speed storage layer is expanded by 300%;

[0030] The compression ratio of cold data is increased to 8:1.

[0031] By automatically adjusting system parameters, the system effectively handles storage pressure during peak periods such as the overlap of stock market openings, improving the system's flexibility and stability, and ensuring normal operation under high concurrency requests.

[0032] In real-time transaction risk monitoring and fraud analysis instances:

[0033] Step 1, continuously monitor the access frequency A and access delay D of the model parameters through the access heat collection module. When a specific model parameter (such as the threshold parameter for identifying fraudulent transactions) is frequently accessed within a very short time, this module will record this access heat information in real time. By monitoring the access heat of model parameters in real time, accurate data basis is provided for subsequent stress analysis and data migration, ensuring that the system can respond to changes in storage pressure in a timely manner.

[0034] Step 2, during the system transaction peak period (such as the overlap period of the opening of major global financial markets), the access frequency A and access delay D will significantly increase, at which time the stress analysis module calculates the storage pressure index P according to the data provided by the access heat collection module, specifically:

[0035] According to the formula , the storage pressure index P is calculated, where and are preset weight coefficients used to adjust the relative importance of access frequency and delay in the calculation of the storage pressure index, satisfying , is the historical peak value of access frequency, used for normalization processing of the current access frequency, is the preset delay threshold, used for normalization processing of the current delay, dynamically calculated based on historical delay data, where is the historical peak value of delay, the formula considers both access frequency and delay to evaluate the storage pressure of the system in real time;

[0036] The preset storage pressure index threshold P th is compared with the calculated storage pressure index P th , and when P>P thWhen the storage pressure index exceeds the preset threshold, the data migration operation instruction is triggered to the migration execution module. By calculating the storage pressure index, the system pressure can be sensed in real time, and the migration operation is triggered accurately when the pressure exceeds the preset threshold, effectively avoiding the problem of delay exceeding caused by storage I / O bottleneck, and ensuring that the system can still maintain millisecond-level response under high concurrency requests.

[0037] wherein the preset weight coefficient and The calculation method is:

[0038] Collect the current concurrency and historical peak concurrency;

[0039] According to the formula , the preset weight coefficient is calculated;

[0040] According to the formula , the preset weight coefficient is calculated. The adjustment of and can ensure that under different load scenarios, the system can reasonably sense the storage pressure according to the changes of access frequency and delay, so as to trigger the data migration operation in time and avoid delay exceeding.

[0041] Third step, after receiving the data migration operation instruction, the migration execution module will intelligently select the model parameters that need to be migrated. It will calculate the heat decay factor and migration weight index of each parameter, generate a priority queue, and only migrate the hot parameters with high frequency access (such as model parameters used for high-frequency transaction analysis) to the high-speed storage layer (such as NVMeSSD). This process ensures that hot data can be quickly accessed under high concurrency requests, thereby reducing the delay. Specifically:

[0042] According to the formula , the heat decay factor H of the parameter is calculated, wherein is the base of natural logarithm, k is the decay coefficient, which is used to control the speed of heat decay, t is the current time, is the last access time, which represents the time when the model parameter was last accessed. The formula is used to calculate the heat decay factor H of the model parameter, that is, the degree of heat decay of the parameter over time. By introducing the decay coefficient k, the system can adjust the speed of heat decay according to the actual demand, ensuring that the heat calculation can accurately reflect the real-time access of the parameter and provide reliable basis for data migration;

[0043] According to the heat decay factor H, and combining the formula , the migration weight index M of the parameter is calculated;

[0044] The calculated migration weight index M is sorted in descending order to generate a priority queue, ensuring that hot parameters with high-frequency access can be migrated to the high-speed storage layer in priority, thereby optimizing system performance.

[0045] Only the top N% of model parameters in the queue are migrated to the high-speed storage layer, where N is the migration ratio, and a dynamic adjustment method is used for real-time updating. By intelligently selecting the model parameters that need to be migrated and generating a priority queue, only the hot parameters with high-frequency access are migrated to the high-speed storage layer, optimizing the utilization of high-speed storage resources, reducing the migration cost, and ensuring fast access to core hot data.

[0046] wherein the dynamic adjustment method of the migration ratio N is:

[0047] Monitor the real-time storage pressure index: obtain the current storage pressure index P calculated by the pressure analysis module, and read the preset pressure threshold level: , , ;

[0048] Hierarchical matching migration ratio: if , set N=15, if , set N=30, if , set N=50;

[0049] Sudden traffic adaptive correction: calculate the concurrent request growth rate , and detect it when the concurrent request growth rate >100% / second, add an emergency increment to the current migration ratio N value, and ensure that the final migration ratio .

[0050] wherein the calculation method of the concurrent request growth rate is:

[0051] Define the time window and sampling points: set the monitoring time window =1 second, mark the continuous sampling points on the time axis: the current time , 1 second ago , 2 seconds ago ;

[0052] Obtain the concurrent request time series data: read the request volume recorded by the concurrent scheduling module: the request volume at time , the request volume at time , the request volume at time ;

[0053] According to the formula , calculate the instantaneous growth rate ;

[0054] According to the formula , calculate the smooth growth rate ;

[0055] According to the formula , output the final concurrent request growth rate , by introducing the smooth growth rate , the system can reduce the impact of instantaneous fluctuations on the calculation results, so as to more accurately judge the growth trend of the request volume.

[0056] Through the cooperative work of the above modules, the system can efficiently process massive transaction streams in the real-time transaction risk monitoring and fraud analysis system of large financial technology companies, realize ultra-low latency and ultra-high throughput real-time analysis, and effectively identify potential fraud patterns, abnormal behaviors and compliance risks.

[0057] In summary, by setting the access heat collection module and stress analysis module, the effect of real-time sensing system stress and accurate triggering of migration is achieved. The access heat collection module continuously monitors the access frequency and delay of model parameters, and the stress analysis module calculates the storage stress index based on historical peak values and dynamic thresholds. When the index exceeds the preset threshold, the migration operation is triggered immediately, solving the problem of delay exceeding the standard caused by storage I / O bottleneck during the peak period of financial transactions, so that the system can still maintain millisecond-level response under tens of thousands of concurrent requests per second, avoiding risk false positives and omissions caused by delay jitter.

[0058] At the same time, when the storage stress index P>0.8P th , the system performs resource allocation through the concurrent scheduling module, specifically:

[0059] Real-time acquisition of the concurrency of the i-th type of request and the storage layer response speed ;

[0060] According to the formula , calculate the load balancing index L, where S max is the maximum value among all storage layer response speeds, and the formula is used to evaluate the load balancing of the system when facing different types of requests. By comparing the storage layer response speed of a specific request with the fastest response speed, the system can determine whether to adjust resource allocation to optimize overall performance and response speed. When the load balancing index exceeds the preset threshold, the system will trigger the corresponding response strategy, such as allocating more bandwidth, activating standby nodes, etc.

[0061] The preset load balancing index threshold L thand the calculated load balancing index L is compared with a preset load balancing index threshold L th When L>L th , the concurrent scheduling module executes a three-level response strategy.

[0062] The calculation method of the storage layer response speed of the ith request is as follows:

[0063] The hit times of the ith request in each storage layer are counted.

[0064] According to the formula , and , the high-speed storage layer weight w1, the medium-speed storage layer weight w2 and the cold storage layer weight w3 are respectively calculated.

[0065] The latency data of the ith request in each storage layer are recorded, including the high-speed storage layer access latency , the medium-speed storage layer access latency and the cold storage layer access latency .

[0066] According to the formula , the storage layer response speed of the ith request is calculated.

[0067] The three-level response strategy is as follows:

[0068] When , the first-level response is triggered, the bandwidth allocated to the ith request is calculated according to the formula , and the bandwidth is allocated according to , wherein is the total bandwidth of the high-speed layer.

[0069] When , the second-level response is triggered, and the standby high-speed storage node is activated.

[0070] When , the third-level response is triggered, the predicted hot spot parameter is preloaded into the memory, and the cold data is downgraded to the HDD layer. Through the hierarchical response strategy, the continuity of the key business is effectively guaranteed. The first-level response dynamically allocates bandwidth to ensure smooth high-priority business; the second-level response activates the standby high-speed node to avoid single-point failure; and the third-level response preloads the predicted hot spot into the memory and downgrades the cold data to the HDD, thereby improving the overall availability and stability of the system.

[0071] ​​​To sum up, through the cooperative work of each module, the multiple effects such as real-time system pressure sensing, accurate triggering of migration, optimization of storage resource utilization, and guarantee of key business continuity are realized, thereby providing strong support for the real-time transaction risk monitoring and fraud analysis system of large financial technology companies.

[0072] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A storage-tiered acceleration system for high-concurrency inference of large models, characterized in that: The system includes an access heat acquisition module, a stress analysis module, a migration execution module, and a heterogeneous storage pool; The access frequency acquisition module is used to collect the access frequency and access latency of the model parameters in real time; The pressure analysis module is used to calculate the storage pressure index based on the data collected by the access popularity acquisition module, and, in conjunction with a preset threshold, determine whether a data migration operation needs to be triggered. Specifically: The storage pressure index is calculated based on access frequency, access latency, historical peak access frequency, and preset weighting coefficients. A preset storage pressure index threshold is set, and the calculated storage pressure index is compared with the preset storage pressure index threshold. When the storage pressure index exceeds the threshold, a data migration operation instruction is triggered to the migration execution module. The heterogeneous storage pool is used to integrate storage media with different performance and cost to form a hierarchical storage structure, which includes a high-speed storage layer, a medium-speed storage layer and a cold storage layer. The high-speed storage layer is used to store frequently accessed hot data, the medium-speed storage layer is used to store data with medium access frequency, and the cold storage layer is used to store data with low access frequency. The migration execution module is used to perform data migration operations according to the data migration operation instructions sent by the stress analysis module, and migrate high-frequency parameters to the high-speed storage layer. When performing data migration operations, the migration execution module performs the following operations for each individual model parameter: The heat decay factor of this parameter is calculated based on the last access time and the current time. The migration weight index of this parameter is calculated based on the heat decay factor and the current access frequency; Sort the calculated migration weight indices in descending order to generate a priority queue; Only a certain percentage of the model parameters in the queue are migrated to the high-speed storage layer. This percentage is the migration ratio, which is dynamically adjusted and updated in real time. The system also includes a concurrent scheduling module, used to perform resource allocation when the storage pressure index exceeds a set threshold, specifically: Real-time acquisition of concurrency levels and storage layer response speed for a specific type of request; Calculate the load balancing index based on concurrency and storage layer response speed; A preset load balancing index threshold is set, and the calculated load balancing index is compared with the preset threshold. When the threshold is exceeded, the concurrent scheduling module executes a three-level response strategy.

2. The storage-layered acceleration system for high-concurrency inference of large models according to claim 1, characterized in that, The method for dynamically adjusting the migration ratio is as follows: Monitor real-time storage pressure index and read preset multi-level pressure threshold levels; Based on the current threshold level of the storage pressure index, match and set the corresponding migration ratio; The system detects the growth rate of concurrent requests. When the growth rate exceeds a set threshold, it adds an emergency increment to the current migration ratio and ensures that the final migration ratio does not exceed the set upper limit.

3. The storage-layered acceleration system for high-concurrency inference of large models according to claim 2, characterized in that, The method for calculating the concurrent request growth rate is as follows: Set the monitoring time window and continuous sampling points; Obtain the number of concurrent requests at each sampling time; Calculate the instantaneous growth rate and the smoothed growth rate, and output the final concurrent request growth rate.

4. The storage-layered acceleration system for high-concurrency inference of large models according to claim 1, characterized in that, The method for calculating the storage layer response speed of a certain type of request is as follows: Count the number of times this type of request hits each storage layer and calculate the weight of each layer; Record the latency data of this type of request in each storage layer; Based on the weights and latency data of each layer, the storage layer response speed for this type of request is calculated.

5. The storage-layered acceleration system for high-concurrency inference of large models according to claim 4, characterized in that, The three-level response strategy is as follows: When the load balancing index is in the first range, a level 1 response is triggered, and bandwidth is dynamically allocated in a specific manner. When the load balancing index is in the second range, a level 2 response is triggered, activating the backup high-speed storage node. When the load balancing index is in the third range, a level 3 response is triggered, which performs preloading of predicted hot spot parameters and degradation of cold data.

6. The storage-layered acceleration system for high-concurrency inference of large models according to claim 1, characterized in that, The calculation method for the preset weighting coefficient is as follows: Collect current concurrency and historical peak concurrency; Based on the relationship between the current concurrency and the historical peak concurrency, the preset weighting coefficient is calculated.

7. The storage-layered acceleration system for high-concurrency inference of large models according to claim 1, characterized in that, The system also includes a data prefetching module, which is used to predict the hot parameter access sequence in the future based on the LSTM neural network. When the predicted access probability exceeds a set threshold, the preloading operation is automatically triggered to migrate the relevant parameters from the cold storage layer to the high-speed storage layer in advance.

8. The storage-layered acceleration system for high-concurrency inference of large models according to claim 1, characterized in that, The system also includes a time-sharing scheduling engine, which has a built-in global financial exchange calendar to track the opening or closing times of each exchange in real time and automatically perform the following operations during overlapping opening periods: Lower the preset storage pressure index threshold; Expand the capacity of the high-speed storage layer; Improve the compression ratio of cold data.

Images