Communication information storage system for information engineering

Abstract

The invention relates to a communication information storage system for information engineering, which belongs to the technical field of information engineering, and comprises a state quantification unit, a dynamic distribution unit, a dynamic distribution unit and a dynamic distribution unit, and is used for generating a dynamic distribution proportion based on a comprehensive risk factor calculated by the state quantification unit, a preset risk trigger threshold value and a preset distribution sensitivity index; and the I / O execution gateway is used for redirecting the newly added I / O request from the main storage layer to the elastic buffer layer according to the dynamic shunting proportion generated by the dynamic shunting unit so as to reduce the instantaneous I / O queue depth and the average I / O delay of the main storage layer, thereby forming a real-time negative feedback regulation closed loop. By accurately quantifying the risk and carrying out dynamic I / O shunting, the peak clipping effect on the load of the main storage layer is realized, so that the system can bear the average load by using economical hardware with lower cost, and meanwhile, the peak load is dealt with by depending on the high-performance elastic buffer layer.

Description

Technical Field

[0001] The present invention relates to the field of information engineering, and in particular to a communication information storage system for information engineering. Background Art

[0002] In modern information engineering, distributed data storage systems such as online transaction processing and real-time analysis platforms generally face the technical contradiction between ensuring business continuity and low latency and controlling total cost of ownership.

[0003] Existing technologies generally adopt two solutions:

[0004] Over-provisioning: By purchasing high-performance storage hardware, such as all-flash arrays, that far exceeds average load requirements, the system is ensured to be able to handle sudden surges in read and write requests for hot data at any time. While this approach effectively ensures that I / O queue depths remain within a safe range, thereby ensuring low latency and high stability for the business, it also has significant drawbacks. Statistics show that over 80% of the time, the resource utilization rate of this expensive hardware is less than 30%, resulting in significant waste and idleness of hardware resources. This directly drives up initial hardware procurement costs and ongoing power and cooling costs, keeping overall costs high.

[0005] Economical hardware deployment: Use economical hardware (such as hybrid disk arrays) that only meets average load requirements to reduce initial procurement costs. However, when experiencing bursts of read and write traffic, this solution can easily fill up the storage node's I / O queue depth in a short period of time, quickly exceeding the hardware's physical limits. For example, the queue depth of conventional SATA / SAS hard drives is typically 32 or 64. This can lead to a large backlog of subsequent I / O requests, a sharp increase in average service latency, and even a chain reaction of request timeouts, potentially causing node downtime and posing a serious challenge to system stability.

[0006] Therefore, existing technical solutions cannot achieve an ideal balance between system performance stability and cost-effectiveness. How to design a storage system that can be deployed with economical hardware costs while intelligently responding to sudden loads and maintaining system stability has become a technical challenge that needs to be solved in this field. Summary of the Invention

[0007] The present invention aims to solve the problems raised in the above background technology and provide a communication information storage system for information engineering, which can significantly reduce the total cost of hardware and operation and maintenance while ensuring the stability of storage node I / O performance.

[0008] The technical solution of the present invention is: a communication information storage system for information engineering, comprising:

[0009] The state quantization unit is used to perform the following steps:

[0010] S11. Obtain the instantaneous I / O queue depth and average I / O latency of a single storage node in the primary storage layer;

[0011] S12. Determine a queue health index based on the instantaneous I / O queue depth, a preset safe queue depth threshold, and a preset maximum queue depth threshold;

[0012] S13. Calculate a comprehensive risk factor based on the queue health index and average I / O latency, and in accordance with a preset weight coefficient and maximum acceptable latency.

[0013] A dynamic diversion unit, configured to generate a dynamic diversion ratio based on the comprehensive risk factor calculated by the state quantification unit, a preset risk trigger threshold, and a preset diversion sensitivity index;

[0014] The I / O execution gateway is used to redirect new I / O requests from the main storage layer to the elastic buffer layer based on the dynamic diversion ratio generated by the dynamic diversion unit, so as to reduce the instantaneous I / O queue depth and average I / O latency of the main storage layer, thereby forming a real-time negative feedback regulation closed loop.

[0015] In this embodiment, the state quantification unit is used to calculate the comprehensive risk factor, and the specific steps include:

[0016] S131. Determine a queue risk component based on the queue health index;

[0017] S132. Determine a delay risk component based on the average I / O delay and the maximum acceptable delay.

[0018] S133. Perform weighted fusion on the queue risk component and the delay risk component to generate the comprehensive risk factor.

[0019] In this embodiment, the queue health index is a normalized indicator, and its value is inversely proportional to the degree to which the instantaneous I / O queue depth exceeds the safe queue depth threshold.

[0020] In this embodiment, the system further includes:

[0021] The intelligent relocation unit is used to perform the following steps:

[0022] S21. Determine the current data migration rate based on the comprehensive risk factors and the maximum migration rate allowed by the system;

[0023] S22: Execute a data migration operation from the elastic buffer layer to the primary storage layer according to the current data migration rate.

[0024] In this embodiment, the system further includes:

[0025] S31, a historical data aggregation unit, is used to perform the following steps:

[0026] S32, obtaining the real-time power consumption of the node;

[0027] S33, obtaining the amount of data written to the elastic buffer layer;

[0028] S34. Quantify the operation and maintenance cost indicators based on real-time power consumption and the amount of written data;

[0029] S35. Aggregate the instantaneous I / O queue depth, comprehensive risk factor, dynamic diversion ratio, and operation and maintenance cost indicators to form a multi-dimensional time series data vector and record it.

[0030] In this embodiment, the system further includes:

[0031] The cost-effectiveness prediction unit is used to predict the expected cost saving rate in a future preset prediction period based on the multi-dimensional time series data vector recorded by the historical data aggregation unit and using a preset prediction model.

[0032] In this embodiment, the system further includes:

[0033] The strategy parameter adaptation unit is used to perform the following steps:

[0034] S41. Determine a cost target deviation based on the expected cost savings rate predicted by the cost-benefit prediction unit and the preset business cost target;

[0035] S42: Based on the cost target deviation, adaptively adjust the security queue depth threshold and the diversion sensitivity index to drive the system operation state toward the business cost target.

[0036] In this embodiment, the adaptive adjustment specifically includes:

[0037] When the cost target deviation indicates that the cost savings target has not been met, increasing the diversion sensitivity index by an amount proportional to the cost target deviation;

[0038] The safe queue depth threshold is lowered by an amount proportional to the cost target deviation.

[0039] In this embodiment,

[0040] The present invention provides a communication information storage system for information engineering through improvements, which has the following improvements and advantages compared with the prior art:

[0041] (1) This invention achieves peak load reduction on the primary storage layer by accurately quantifying risk and performing dynamic I / O diversion. This allows the system to use lower-cost, economical hardware to carry the average load while relying on a high-performance, elastic buffer layer to handle peak loads. This avoids the huge cost waste of over-configuring expensive hardware to handle occasional peaks, while also resolving the problem of economical hardware being prone to crashing when faced with peak loads, thereby achieving an excellent balance between system stability and cost-effectiveness.

[0042] (2) The present invention constructs a closed-loop real-time negative feedback regulation system. The state quantization unit serves as the system's perception module. It converts the original, multi-dimensional performance indicators (queue depth, latency) into a standardized, single comprehensive risk factor through a mathematical model, accurately characterizing the load pressure and performance deterioration risk of the current storage node. The dynamic diversion unit serves as the system's decision-making module. According to the quantified risk size, it calculates a specific, executable dynamic diversion ratio. Finally, the I / O execution gateway serves as the system's execution module. According to this ratio, it accurately diverts some of the newly added I / O requests from the main storage layer with greater pressure to the high-performance elastic buffer layer. This execution action directly reduces the load on the main storage layer, thereby returning its I / O queue depth and latency to a healthy level, forming a stable, self-regulating closed loop. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will be further explained below in conjunction with the accompanying drawings and Examples:

[0044] Figure 1 It is a flow chart of the system of the present invention. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific embodiments.

[0046] Example 1:

[0047] See also Figure 1 The present invention provides a technical solution for a communication information storage system for information engineering: a communication information storage system for information engineering, the specific steps of which include:

[0048] A communication information storage system for information engineering, comprising:

[0049] The state quantization unit is used to perform the following steps:

[0050] S11. Obtain the instantaneous I / O queue depth and average I / O latency of a single storage node in the primary storage layer;

[0051] Instantaneous I / O queue depth (Q depth) refers to the number of I / O requests waiting to be processed in the driver queue of the primary storage tier storage device at a specific moment. This data can be obtained by a monitoring agent deployed on each storage node by frequently querying the operating system kernel interface (for example, once per second). For example, in Linux, this data can be obtained by reading the / proc / diskstats file.

[0052] Average I / O latency (L node ) refers to the average time taken by all completed I / O requests from submission to completion in the most recent sampling period. This data is calculated by the monitoring agent by calculating the difference between the initiation time and the completion response time of the I / O request.

[0053] S12. Determine a queue health index based on the instantaneous I / O queue depth, a preset safe queue depth threshold, and a preset maximum queue depth threshold;

[0054] Determine the queue health index (H queue )

[0055] This step converts the original queue depth value into a normalized, inverse health indicator. The calculation formula is as follows:

[0056]

[0057] Parameter Description:H queue : Queue health index, a dimensionless normalized value in the range [0,1]. depth : Instantaneous I / O queue depth. Q safe : The preset safe queue depth threshold is initially configured by the system administrator based on hardware performance and business delay tolerance.

[0058] Q max : The maximum queue depth set at the node physical or software level, which is a technical specification parameter of the hardware or driver.

[0059] The queue health index is a normalized metric whose value is inversely proportional to the degree to which the instantaneous I / O queue depth exceeds the safety threshold. This design converts queue depths on different hardware into a unified measurement scale, providing a stable and comparable input for subsequent risk calculations.

[0060] S13. Calculate a comprehensive risk factor based on the queue health index and average I / O latency, and according to the preset weight coefficient and maximum acceptable latency. This step combines queue congestion trends and existing latency deterioration to generate a comprehensive risk factor.

[0061] A dynamic diversion unit, configured to generate a dynamic diversion ratio based on the comprehensive risk factor calculated by the state quantification unit, a preset risk trigger threshold, and a preset diversion sensitivity index;

[0062] Dynamic diversion unit: based on R instability Calculate the dynamic diversion ratio P shunt .

[0063] The calculation formula is as follows:

[0064]

[0065] Parameter Description: shunt R: Dynamic diversion ratio, dimensionless, value range [0,1]. instability : Comprehensive risk factor. R trigger : The dimensionless threshold for activating I / O offloading risk. This threshold is a benchmark value that can be set empirically by those skilled in the art. For example, it can be set to the comprehensive risk factor value corresponding to the inflection point where system I / O latency begins to show nonlinear growth in historical stress testing. A value between 0.3 and 0.5 is generally recommended.

[0066] γ: Diversion sensitivity index, a constant greater than 1. The design with power index γ>1 makes the response nonlinear: when the risk is low, R instability Less than R trigger , the diversion ratio is close to 0, and the strategy is moderate; when the risk is high, R instability Significantly exceeds R trigger , the diversion ratio will quickly approach 1, indicating a decisive policy. A higher γ value results in a more aggressive response. This value can be adjusted based on the service's tolerance for burst traffic, with a typical value of 2.

[0067] The I / O execution gateway is used to redirect new I / O requests from the primary storage layer to the elastic buffer layer based on the dynamic diversion ratio generated by the dynamic diversion unit, so as to reduce the instantaneous I / O queue depth and average I / O latency of the primary storage layer, thereby forming a real-time negative feedback regulation loop. shunt value, redirecting the corresponding proportion of requests to the elastic buffer layer to complete the negative feedback loop.

[0068] In this embodiment, the state quantification unit is used to calculate the comprehensive risk factor, and the specific steps include:

[0069] S131. Determine a queue risk component based on the queue health index;

[0070] S132. Determine a delay risk component based on the average I / O delay and the maximum acceptable delay.

[0071] S133. Perform weighted fusion on the queue risk component and the delay risk component to generate the comprehensive risk factor.

[0072] The calculation formula is as follows:

[0073] Parameter Description: R instability : Comprehensive risk factor, dimensionless, ranging from [0,1]. α, β: Weight coefficients, dimensionless, and satisfying α+β=1. L node : Average I / O latency, in milliseconds (ms); The configuration of these two coefficients allows technicians in this field to adjust them according to specific business needs. For example, for businesses that are extremely sensitive to latency (such as online transaction processing), a higher weight should be given to latency risk, and β>α can be set (such as α=0.4, β=0.6). For businesses that prioritize throughput (such as big data analysis), the weight of queue risk can be appropriately increased, and α>β can be set (such as α=0.7, β=0.3). max : The maximum acceptable latency specified in the Service Level Agreement (SLA), in milliseconds (ms).

[0074] By weighting and integrating queue risk (potential congestion) and delay risk (actual performance deterioration), system decisions can be both preventive and reactive, which is more comprehensive and accurate than any single indicator.

[0075] In this embodiment, the queue health index is a normalized indicator, and its value is inversely proportional to the degree to which the instantaneous I / O queue depth exceeds the safe queue depth threshold.

[0076] In this embodiment, the system further includes:

[0077] The intelligent relocation unit is used to perform the following steps:

[0078] S21. Determine the current data migration rate based on the comprehensive risk factors and the maximum migration rate allowed by the system;

[0079] S22, and execute the data migration operation from the elastic buffer layer to the main storage layer according to the current data migration rate. migrate rate, and initiates data migration from the elastic buffer layer to the primary storage layer.

[0080] Determine the current data retrieval rate (V migrate )

[0081] The calculation formula is as follows:

[0082] V migrate =V max ·(1-R instability )

[0083] Parameter Description: V migrate : Real-time data migration rate, in MB / s. V max : The maximum repatriation rate allowed by the system, in MB / s. instability : Comprehensive risk factor of the node.

[0084] Ensure that the migration is performed only when the system load is low (R instability close to 0) to avoid adding extra burden to busy systems.

[0085] It implements intelligent valley-filling management of elastic buffer layer resources, ensuring that buffer resources are promptly recovered when the system is idle, while ensuring that the migration process does not interfere with normal business.

[0086] Example 2: Cost-benefit prediction and strategic adaptation based on historical data. In this example, the system further includes:

[0087] S31, a historical data aggregation unit, is used to perform the following steps:

[0088] S32. Acquire the real-time power consumption of the node through an out-of-band management interface (such as IPMI) of the server.

[0089] S33: Obtain the amount of data written to the elastic buffer layer; and perform statistics by the I / O execution gateway.

[0090] S34. Based on real-time power consumption and the amount of written data, quantify the operation and maintenance cost indicators; calculate through a quantitative model that uniformly converts physical consumption and resource loss into standardized cost rate indicators.

[0091] Quantified operation and maintenance cost rate indicator C ops_rate (t) Calculation process:

[0092]

[0093] C ops_rate (t): Operation and maintenance cost rate around time point t, in normalized cost units / second.

[0094] W power : Power consumption cost weight coefficient, the unit is normalized cost unit / watt-hour or normalized cost unit / joule.

[0095] The average power consumption during the time interval Δt, in watts (W).

[0096] W wear : SSD write wear cost weight coefficient, the unit is "normalized cost unit / byte".

[0097] ΔWbytes_T2 (t): The amount of data written to the elastic buffer layer within the time interval Δt.

[0098] The average write rate in the time interval Δt, in bytes per second (B / s).

[0099] S35. Aggregate the instantaneous I / O queue depth, comprehensive risk factor, dynamic diversion ratio, and operation and maintenance cost indicators to form a multi-dimensional time series data vector and record it.

[0100] Aggregate to form a multidimensional time series data vector

[0101] At each time point t, all relevant data are aggregated into a vector and recorded:

[0102] By establishing a comprehensive historical data set that relates system status, control behavior, and operating costs, memory capabilities are introduced into the system, which is the fundamental basis for subsequent performance analysis, cost-benefit prediction, and strategy optimization.

[0103] In this embodiment, the system further includes:

[0104] The cost-effectiveness prediction unit is used to predict the expected cost saving rate in a future preset prediction period based on the multi-dimensional time series data vector recorded by the historical data aggregation unit and using a preset prediction model.

[0105] This unit uses historical data to train a preset prediction model, such as a convolutional neural network (CNN), to learn the mapping relationship between load pattern -> control behavior -> cost results.

[0106] In this embodiment, the prediction model is specifically a convolutional neural network (CNN), and its implementation details are as follows:

[0107] Model input: A two-dimensional tensor representing the system state and behavior over the past N time steps. The tensor has dimensions N × M, where N is the number of time steps. For example, N = 60 represents data from the past 60 seconds, and M is the feature dimension for each time step. This M-dimensional feature vector specifically includes the normalized instantaneous I / O queue depth, comprehensive risk factor, dynamic offload ratio, and O&M cost rate indicator.

[0108] Model Architecture: The CNN model consists of the following main layers:

[0109] One-dimensional convolution layer Conv1D: contains 16 filters with a convolution kernel size of 3, which is used to extract local, short-term temporal features from the time series.

[0110] Activation layer: uses ReLU activation function.

[0111] Pooling layer: MaxPooling is used to reduce the data dimension and retain the most significant features.

[0112] Fully connected layer: Contains two fully connected layers with 32 and 16 neurons respectively, which are used to perform nonlinear combination of the extracted features.

[0113] Output layer: A single neuron output layer that does not use an activation function (or uses linear activation) and outputs a scalar value.

[0114] Model output: predicted expected cost savings rate η cost_pred , which is a dimensionless scalar value.

[0115] Model training: The model is trained offline using supervised learning. The training dataset comes from the historical data aggregation unit and is labeled with the actual cost savings rates observed in actual operations. The training objective is to minimize the mean squared error between the predicted and true values.

[0116] By introducing this prediction unit, the system becomes forward-looking and can predict future economic benefits before adjusting strategies, which is a prerequisite for achieving proactive, goal-driven strategic optimization.

[0117] This model is used to predict the future of a preset prediction period (T predict ) within the expected cost saving rate (η cost_pred ).

[0118] Making the system forward-looking and able to predict the future economic benefits of its current operating strategy is a prerequisite for achieving proactive, goal-driven strategic optimization.

[0119] In this embodiment, the system further includes:

[0120] The strategy parameter adaptation unit is used to perform the following steps:

[0121] S41. Determine a cost target deviation based on the expected cost savings rate predicted by the cost-benefit prediction unit and the preset business cost target;

[0122] Determine the cost target deviation (Δ c )

[0123] Δ c =η target -η cost_pred

[0124] Parameter Description:Δ c : Cost target deviation factor. Δ c >0 means the cost savings have not reached the target. target: The cost reduction target pursued by the system, for example, 0.4. cost_pred : Forecasted expected cost savings rate.

[0125] S42: Based on the cost target deviation, adaptively adjust the security queue depth threshold and the diversion sensitivity index to drive the system operation state toward the business cost target.

[0126] When Δ c When >0, the system determines that the current strategy is too conservative and needs to use the elastic buffer more actively. The adjustment strategy is as follows:

[0127] Increase the shunt sensitivity index:

[0128] γ new =γ current ·(1+k1·Δ c )

[0129] Lower the safe queue depth threshold:

[0130] Q safe,new =Q safe,current ·(1-k2·Δ c )

[0131] Specifically, γ new ,Q safe,new : The new parameter value after adjustment. γ current ,Q safe,current : The parameter value currently being used. k1, k2: Adjustment coefficients.

[0132] This unit builds a strategic feedback loop for the system, directly linking high-level business goals with underlying technical operating parameters, and implements automated optimization and adjustment, enabling the system to self-evolve and continuously maintain the optimal balance between performance and cost.

[0133] In this embodiment, the adaptive adjustment specifically includes:

[0134] When the cost target deviation indicates that the cost savings target has not been met, increasing the diversion sensitivity index by an amount proportional to the cost target deviation;

[0135] The safe queue depth threshold is lowered by an amount proportional to the cost target deviation.

[0136] This embodiment builds a multi-layered, adaptive, intelligent closed-loop control system: it includes not only a fast tactical closed-loop for real-time load adjustment, but also a slow strategic closed-loop for long-term policy optimization. This system decomposes abstract business objectives layer by layer, ultimately translating them into specific, automated adjustments to underlying technical parameters, significantly improving the system's automated operations and maintenance capabilities.

[0137] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A communication information storage system for information engineering, characterized in that: include: The state quantization unit is used to perform the following steps: S11. Obtain the instantaneous I / O queue depth and average I / O latency of a single storage node in the primary storage layer; S12. Determine a queue health index based on the instantaneous I / O queue depth, a preset safe queue depth threshold, and a preset maximum queue depth threshold; S13. Calculate a comprehensive risk factor based on the queue health index and average I / O latency, and in accordance with a preset weight coefficient and maximum acceptable latency. A dynamic diversion unit, configured to generate a dynamic diversion ratio based on the comprehensive risk factor calculated by the state quantification unit, a preset risk trigger threshold, and a preset diversion sensitivity index; The I / O execution gateway is used to redirect new I / O requests from the main storage layer to the elastic buffer layer based on the dynamic diversion ratio generated by the dynamic diversion unit, so as to reduce the instantaneous I / O queue depth and average I / O latency of the main storage layer, thereby forming a real-time negative feedback regulation closed loop.

2. A communication information storage system for information engineering according to claim 1, characterized in that: The state quantification unit is used to calculate the comprehensive risk factor, and the specific steps include: S131. Determine a queue risk component based on the queue health index; S132. Determine a delay risk component based on the average I / O delay and the maximum acceptable delay; S133. Perform weighted fusion on the queue risk component and the delay risk component to generate the comprehensive risk factor.

3. A communication information storage system for information engineering according to claim 1, characterized in that: The queue health index is a normalized indicator, and its value is inversely proportional to the degree to which the instantaneous I / O queue depth exceeds the safe queue depth threshold.

4. A communication information storage system for information engineering according to claim 3, characterized in that: The system further comprises: The intelligent relocation unit is used to perform the following steps: S21. Determine the current data migration rate based on the comprehensive risk factors and the maximum migration rate allowed by the system; S22: Execute a data migration operation from the elastic buffer layer to the primary storage layer according to the current data migration rate.

5. A communication information storage system for information engineering according to claim 4, characterized in that: The system further comprises: S31, a historical data aggregation unit, is used to perform the following steps: S32, obtaining the real-time power consumption of the node; S33, obtaining the amount of data written to the elastic buffer layer; S34. Quantify the operation and maintenance cost indicators based on real-time power consumption and the amount of written data; S35, aggregate instantaneous I / O queue depth, comprehensive risk factors, dynamic diversion ratio and operation and maintenance cost indicators to form a multi-dimensional time series data vector and record it.

6. A communication information storage system for information engineering according to claim 5, characterized in that: The system further comprises: The cost-effectiveness prediction unit is used to predict the expected cost saving rate in a future preset prediction period based on the multi-dimensional time series data vector recorded by the historical data aggregation unit and using a preset prediction model.

7. A communication information storage system for information engineering according to claim 6, characterized in that: The system further comprises: The strategy parameter adaptation unit is used to perform the following steps: S41. Determine a cost target deviation based on the expected cost savings rate predicted by the cost-benefit prediction unit and the preset business cost target; S42: Based on the cost target deviation, adaptively adjust the security queue depth threshold and the diversion sensitivity index to drive the system operation state toward the business cost target.

8. A communication information storage system for information engineering according to claim 7, characterized in that: The adaptive adjustment specifically includes: When the cost target deviation indicates that the cost savings target has not been met, increasing the diversion sensitivity index by an amount proportional to the cost target deviation; The safe queue depth threshold is lowered by an amount proportional to the cost target deviation.

Images