A processing system for improving server data storage speed

Abstract

The application relates to the technical field of server data storage, and discloses a processing system for improving server data storage speed, which comprises a data blocking module, a parallel optimization module, a performance monitoring module, a strategy matching module and a storage execution module, and can comprise a load self-learning module. The data blocking module blocks data streams according to a dynamic strategy and generates a distribution queue; the parallel optimization module generates an optimal storage node combination and a scheduling strategy according to the queue; the performance monitoring module collects real-time performance data and screens for abnormalities; the strategy matching module determines a target adjustment rule through multidimensional matching; the storage execution module drives a node to correct a path; and the load self-learning module optimizes a scheduling factor according to feedback data. The system realizes dynamic blocking, adaptive scheduling and real-time optimization, improves storage efficiency and stability, and is suitable for a server data storage scene.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of server data storage, and particularly relates to a processing system for improving server data storage speed. BACKGROUND

[0002] In the cloud computing and big data era, server data storage is facing the severe challenges of explosive growth of data volume and continuous improvement of storage performance. The traditional server data storage system has many defects that are difficult to overcome when processing large-scale data streams, which seriously restricts the data storage efficiency and the overall performance of the system. From the perspective of data blocking and distribution mechanism, the traditional system usually adopts a fixed blocking strategy, which cannot be dynamically adjusted according to the type and characteristics of the data stream. For new data stream types, there is a lack of effective blocking parameter collection and strategy calibration mechanism, which leads to unreasonable data block division and affects the efficiency of subsequent storage operations. For historical data stream types, the historical blocking data set and storage performance change curve and other information cannot be fully utilized, making it difficult to optimize the blocking processing, and the matching degree of data blocks and storage nodes is low, increasing the delay of data transmission and storage.

[0003] In the state management and scheduling of the storage nodes, the traditional system is not comprehensive and dynamic in monitoring and utilizing the state of the storage nodes. Only some basic static state information of the storage nodes can be obtained, such as the node available bandwidth, the current IOPS capacity value and the storage space remaining threshold, but the dynamic load coefficients such as the bandwidth allocation weight, the IOPS priority coefficient and the space occupation adjustment factor are ignored. This makes the generation of the data block distribution queue lack scientificity and dynamic adaptability, and cannot be reasonably adjusted according to the real-time storage node state data, which easily leads to unbalanced load of the storage nodes, part of the nodes being overloaded and part of the nodes being idle, seriously affecting the data storage speed and the stability of the system.

[0004] In the generation of the parallel storage scheduling strategy, the scheduling rule library of the traditional system is relatively fixed and single, and lacks adaptive adjustment capability. When facing different data block distribution queues, the system cannot perform effective pattern matching and priority adjustment, and the generated scheduling strategy cannot well adapt to the real-time changing storage requirements. For example, in terms of data block transmission rules, node selection conditions and fault switching mechanism, the system lacks flexibility and optimization capability, and easily has problems such as high response delay, large transmission rate fluctuation and poor IOPS stability in the data transmission process, resulting in low data storage efficiency.

[0005] Performance monitoring and strategy adjustment mechanisms are another weak link in traditional systems. Traditional performance monitoring methods cannot obtain real-time storage performance data streams from storage nodes in a timely and accurate manner, and the screening and processing of abnormal performance data is not timely or effective enough. When storage performance is abnormal, the corresponding scheduling rules cannot be quickly and accurately identified and adjusted, resulting in problems that cannot be resolved in a timely manner, further affecting data storage speed and system performance. In addition, the lack of effective self-learning and optimization mechanisms makes it impossible to continuously optimize scheduling strategies based on historical storage data and feedback information, making it difficult for system performance to improve over time. Summary of the Invention

[0006] The purpose of this invention is to provide a processing system that improves the data storage speed of a server, thereby solving the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a processing system for improving server data storage speed, the system comprising:

[0008] The data segmentation module is used to receive the data stream to be stored, divide the data stream into multiple data blocks according to a predefined dynamic segmentation strategy, and generate a data block allocation queue based on the storage node status database.

[0009] The parallel optimization module, based on the data block allocation queue, processes and generates the optimal combination of storage nodes for each data block and generates a parallel storage scheduling strategy.

[0010] The performance monitoring module is used to obtain real-time storage performance data streams through performance probes deployed on storage nodes, filter out abnormal performance data that meets the dynamic adjustment range, and send them to the policy matching module.

[0011] The strategy matching module is used to perform multi-dimensional matching between abnormal performance data and each scheduling rule item in the parallel storage scheduling strategy, generate the matching degree between real-time performance data and each scheduling rule item, and select the scheduling rule corresponding to the highest matching degree as the target adjustment rule.

[0012] The storage execution module is used to receive the target adjustment rules and call the preset scheduling adjustment protocol in the storage node status database to drive the storage node to perform data storage path correction operations.

[0013] Preferably, the process of dividing the data stream into multiple data blocks according to a predefined dynamic partitioning strategy is as follows:

[0014] The unique identifier of the data stream is identified. If it is a new data stream type, the basic block parameters are collected, including the initial block size range, priority weight factor and redundancy configuration baseline value. Based on the initial data, block feature nodes are established and policy calibration is performed to generate a data block allocation queue.

[0015] If it is a historical data stream type, extract the historical chunk data set of this type and store the performance change curve, the historical chunk data set includes block size adjustment frequency, node load balancing record and transmission delay fluctuation value, and is marked as the initial parameter set of the current chunk processing.

[0016] Preferably, the data block allocation queue is generated based on the storage node state database, specifically including:

[0017] The initial state set of each storage node and the dynamic load coefficient set are extracted from the storage node state database, and the initial state set includes: node available bandwidth range, current IOPS capacity value and storage space remaining threshold;

[0018] The dynamic load coefficient set includes bandwidth allocation weight, IOPS priority coefficient and space occupation adjustment factor;

[0019] Based on real-time storage node state data, the initial state set of each storage node is dynamically adjusted, and the adjusted state set is recorded as the data block allocation queue;

[0020] The data block allocation queue includes the bandwidth dynamic allocation value, IOPS optimization control value and space occupation warning point of each node.

[0021] Preferably, the optimal storage node combination of each data block is generated based on the data block allocation queue, and a parallel storage scheduling strategy is generated, specifically including:

[0022] According to the mode matching of the data block allocation queue and the preset storage scheduling rule library, the scheduling rule adjustment priority queue is determined;

[0023] Based on the adjustment priority queue, an adaptive scheduling strategy containing node selection conditions, data block transmission rules and fault switching mechanism is generated.

[0024] Preferably, the real-time storage performance data stream is obtained by deploying a performance probe on the storage node, specifically including:

[0025] The real-time storage performance data stream of the data block transmission process is monitored, including node response delay value, data transmission rate fluctuation and IOPS stability index;

[0026] When the real-time storage performance data stream exceeds the preset qualified performance range, the abnormality flag is activated and the

[0027] The performance data of the abnormal period is taken as the effective monitoring data.

[0028] Preferably, the matching degree of the real-time performance data and each scheduling rule item is generated, specifically including:

[0029] The response delay increment, the transmission rate change gradient and the IOPS fluctuation amplitude in the abnormal performance data are respectively subtracted from the bandwidth dynamic allocation value, the IOPS optimization control value and the space occupation early warning point of each scheduling rule item;

[0030] The matching degree index of the real-time performance data and each scheduling rule item is generated based on the subtraction calculation result.

[0031] Preferably, the scheduling rule corresponding to the highest matching degree is selected as the target adjustment rule, and the specific operation includes:

[0032] A matching degree sorting list of each scheduling rule item is established, and the rule item corresponding to the first matching degree in the list is selected.

[0033] If the first matching degree is lower than a preset adjustment trigger threshold, a backup scheduling rule set is called and the matching degree calculation is performed again.

[0034] Preferably, the matching degree index of the real-time performance data and each scheduling rule item is generated based on the subtraction calculation result, and the specific processing process is:

[0035] The bandwidth difference, the IOPS deviation and the space occupation difference are standardized by using a multi-dimensional matching algorithm to generate a scheduling rule matching degree value in the range of 0-100.

[0036] The closer the matching degree value is to 100, the stronger the adaptability of the real-time performance data to the scheduling rule.

[0037] Preferably, the system further includes a load self-learning module, and the specific operation includes:

[0038] The data block storage feedback data after each storage path correction is recorded, including the actual transmission completion time, the node load balancing degree and the data integrity check result.

[0039] The feedback data is reversely verified with the parallel storage scheduling strategy, a scheduling correction factor is generated and is updated to the dynamic load coefficient set of the storage node state database.

[0040] Preferably, the scheduling correction factor is generated, and the specific operation includes:

[0041] Based on the deviation degree of the feedback data from the expected performance target, a bandwidth optimization factor, an IOPS adjustment factor and a space management weight are calculated.

[0042] The historical correction factors are dynamically smoothed by using an exponential weighted average algorithm to generate a new dynamic load coefficient set.

[0043] Compared with the prior art, the present application has the following beneficial effects:

[0044] The data chunking module realizes accurate application of a dynamic chunking strategy by distinguishing between new data streams and historical data stream types. For new data streams, basic chunking parameters are collected and a chunking feature node is established for strategy calibration, ensuring the rationality of the first chunking. For historical data streams, a historical chunking dataset and a storage performance change curve are used as an initial parameter set, enabling the chunking process to be optimized based on past experience and improving the fit between data block division and data stream characteristics, laying a foundation for efficient subsequent storage operations.

[0045] The mechanism for generating a data block allocation queue based on a storage node state database comprehensively integrates an initial state set and a dynamic load coefficient set. Not only does it consider basic states such as node available bandwidth and IOPS capacity, but it also introduces dynamic factors such as bandwidth allocation weight and IOPS priority coefficient, and dynamically adjusts them according to real-time states, enabling the allocation queue to reflect node load conditions in real time and effectively balance bandwidth, IOPS, and storage space utilization of each node, avoiding node overload or idling and improving overall storage resource utilization.

[0046] The parallel optimization module generates an adaptive scheduling strategy containing node selection conditions, transmission rules, and fault switching mechanisms through pattern matching with a pre-set storage scheduling rule library and priority adjustment. This strategy can dynamically optimize storage paths according to the actual situation of the data block allocation queue, achieve optimal node combination during data transmission, shorten data block transmission time, and enhance system fault tolerance through a fault switching mechanism, ensuring the stability and reliability of the storage process.

[0047] The performance monitoring module uses performance probes deployed on storage nodes to obtain performance data streams containing response delay, transmission rate fluctuation, and IOPS stability in real time and accurately filter abnormal data. The strategy matching module uses a multi-dimensional matching algorithm to perform difference calculation and matching degree analysis between abnormal data and scheduling rule items, quickly locates the most suitable target adjustment rule, realizes real-time response and accurate adjustment of storage performance abnormalities, timely corrects storage paths, restores system performance, and reduces the impact of performance fluctuations on storage speed.

[0048] The load self-learning module records feedback data such as actual transmission completion time and node load balancing degree after storage path correction, and performs reverse verification with the scheduling strategy to generate scheduling correction factors to update the dynamic load coefficient set. This self-learning mechanism based on historical data enables the system to continuously accumulate experience, optimize scheduling strategies and node state management, form a "monitoring-adjustment-learning-optimization" closed loop, and continuously improve the performance and adaptability of the storage system, enabling it to evolve and adapt to increasingly complex storage demands in the long run. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1The working principle diagram of the processing system for improving the server data storage speed according to the present application;

[0050] Figure 2 The design diagram for the dynamic block strategy execution;

[0051] Figure 3 The design diagram for the data block allocation queue generation;

[0052] Figure 4 The design diagram for the real-time storage performance monitoring;

[0053] Figure 5 The design diagram for the target adjustment rule selection. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0055] Please refer to Figures 1-5 The present application relates to a processing system for improving the server data storage speed, which comprises a data block module, a parallel optimization module, a performance monitoring module, a strategy matching module and a storage execution module. Specifically, the following steps are included:

[0056] The data block module is used for receiving a data stream to be stored, dividing the data stream into a plurality of data blocks according to a predefined dynamic block strategy, and generating a data block allocation queue based on a storage node state database;

[0057] The parallel optimization module is used for processing the optimal storage node combination of each data block and generating a parallel storage scheduling strategy based on the data block allocation queue;

[0058] The performance monitoring module is used for obtaining a real-time storage performance data stream through a performance probe deployed on a storage node, screening out abnormal performance data meeting a dynamic adjustment range and sending the abnormal performance data to the strategy matching module;

[0059] The strategy matching module is used for performing multi-dimensional matching between the abnormal performance data and each scheduling rule item in the parallel storage scheduling strategy, generating a matching degree of the real-time performance data and each scheduling rule item, and selecting a scheduling rule corresponding to the highest matching degree as a target adjustment rule;

[0060] The storage execution module is used for receiving the target adjustment rule and calling a preset scheduling adjustment protocol in the storage node state database, and driving the storage node to perform a data storage path correction operation.

[0061] The application will be further described in connection with examples 1 to 5:

[0062] Example 1:

[0063] When the data blocking module in the system performs the data blocking operation, the data stream type needs to be identified, and different processing methods are adopted for different types of data streams to achieve reasonable division of data streams and generation of data block allocation queues.

[0064] When processing the data stream, the data blocking module first identifies the unique identifier of the data stream. The unique identifier is the key basis for distinguishing different data stream types, and by analyzing and judging it, it can be determined whether the current processed data stream is a new data stream type or a historical data stream type.

[0065] When the unique identifier of the data stream corresponds to a new data stream type, the data blocking module will start a series of processing procedures for new type data. At this time, the module will collect basic blocking parameters, which are crucial for the blocking processing of new data streams. Among them, the initial block size range determines the initial division scale of the data block, which provides a basic size reference for subsequent data blocking; the priority weight factor is used to measure the importance of the data stream in the storage process, and different priority data streams will be treated differently in subsequent storage resource allocation and other links. The priority weight factor is a quantitative index for measuring the priority level of the new data stream in storage resource allocation and transmission scheduling, and its value is determined based on the business attributes of the data stream, data timeliness requirements and system resource occupation requirements. It is used to prioritize high importance data streams in concurrent data stream storage to ensure storage efficiency; the redundancy configuration reference value involves the redundancy strategy of data storage, ensuring the reliability and fault tolerance of data in the storage process. The redundancy configuration reference value is the basic reference standard for data redundancy storage, which is used to clearly define the minimum redundancy guarantee requirements that the type of data stream needs to meet in the storage process. Specifically, it corresponds to the basic number of data backup copies, and its core function is to balance the reliability of data storage and the storage resource occupancy rate, avoiding resource waste caused by excessive redundancy or data loss risk caused by insufficient redundancy.

[0066] After collecting the basic chunking parameters, the data chunking module will establish a chunking feature node based on the first received data. This chunking feature node is an abstraction and record of the characteristics of the new data stream, which contains various key information of the data stream during the first processing. After establishing the chunking feature node, the module will perform a strategy calibration operation. Strategy calibration is to adjust and optimize the previously set chunking strategy according to the actual situation of the first data, so that it is more in line with the characteristics of the new data stream, thereby ensuring the accuracy and effectiveness of subsequent chunking operations. After strategy calibration, the data chunking module finally generates a data block allocation queue, which will provide an important basis for subsequent data storage allocation.

[0067] When the identified is a historical data stream type, the data chunking module handles it differently. At this time, the module will extract the historical chunking data set of this type and the storage performance change curve. The historical chunking data set contains rich information, among which the block size adjustment frequency reflects the adjustment of the block size in the past storage process of this type of data stream. By understanding this frequency, a reference can be provided for the current block size setting, making it more in line with the storage habits of this type of data; the node load balancing record records the load of each storage node in the past storage process, which helps to better achieve the balanced allocation of node load in the current chunking process; the transmission delay fluctuation value reflects the delay change in the data transmission process, which is the deviation data and change trend record between the actual delay of each data transmission in the historical storage process of the historical data stream type data stream and the average transmission delay of this data stream, specifically reflecting the stability of data transmission delay. If the fluctuation value is small, it means that the transmission delay of the historical data stream type data stream is relatively stable, and the chunking strategy can remain relatively fixed; if the fluctuation value is large, it indicates that the block size or storage node needs to be dynamically adjusted in the current chunking process to reduce the impact of delay fluctuation on storage speed. Through the analysis of this value, appropriate measures can be taken to reduce the transmission delay in the current chunking process.

[0068] The storage performance change curve shows the performance change trend of this type of data stream in the storage process from a macro perspective. Through the analysis of this curve, the performance of the storage at different stages can be understood, providing a more comprehensive reference for the current chunking process and storage strategy formulation.

[0069] After extracting the historical chunking data set and the storage performance change curve, the data chunking module will mark the block size adjustment frequency, node load balancing record, and transmission delay fluctuation value in the historical chunking data set as the initial parameter set for the current chunking process. These initial parameter sets are based on the storage experience and data of this type of data stream, and using them as the initial parameters for the current chunking process can make the chunking process more in line with the characteristics of this type of data, improving the rationality and efficiency of chunking.

[0070] Through the distinguishing processing of the new data stream type and the historical data stream type, the data block module can take corresponding block parameter collection, feature node establishment, strategy calibration and initial parameter setting operations according to the characteristics of different data streams, so as to realize the reasonable division of the data stream and generate an accurate data block allocation queue.

[0071] Embodiment 2:

[0072] When the data block module generates the data block allocation queue based on the storage node state database, a series of operations such as extracting related data from the database and dynamically adjusting the state set need to be completed to ensure that the generated allocation queue can accurately reflect the real-time state of the storage node and provide an effective basis for subsequent data storage.

[0073] The data block module will extract the initial state set and the dynamic load coefficient set of each storage node from the storage node state database. The storage node state database is an important data structure that stores all the relevant state information of the storage nodes, which contains the state records and load-related parameters of each node at different times.

[0074] The initial state set covers key information such as the node available bandwidth range, the current IOPS capacity value and the storage space remaining threshold. The node available bandwidth range represents the bandwidth interval that the storage node can use for data transmission in the current state, which reflects the data transmission capability of the node; the current IOPS capacity value represents the number of input / output operations that the node can handle per unit time, which is one of the important indicators to measure the performance of the node; the storage space remaining threshold is used to determine whether the remaining storage space of the node reaches the degree that needs to be warned or limited for storage, to ensure that the node has enough space to store new data.

[0075] The dynamic load coefficient set includes bandwidth allocation weight, IOPS priority coefficient and space occupation adjustment factor. The bandwidth allocation weight is used to determine the relative importance of each node in the bandwidth resource allocation during data storage; the IOPS priority coefficient determines the priority order of each node when processing IOPS operations; the space occupation adjustment factor is used to adjust the related parameters according to the storage space occupation of the node to achieve more reasonable allocation of storage resources.

[0076] After extracting the initial state set and the dynamic load coefficient set, the data block module will dynamically adjust the initial state set of each storage node based on the real-time storage node state data. The real-time storage node state data is collected in real time by the monitoring components deployed on each storage node, which can accurately reflect the current actual running state of the node.

[0077] In the dynamic adjustment process, the module will comprehensively consider various factors. For example, for the node available bandwidth range, real-time adjustment will be made according to the current node network load, the occupation of other data transmission tasks, etc.; for the current IOPS capacity value, re-evaluation will be made in combination with the current processor load, memory usage, etc. of the node; for the storage space remaining threshold, update will be made according to the latest storage space usage of the node.

[0078] The state set after dynamic adjustment is the data block allocation queue. The data block allocation queue contains key information such as bandwidth dynamic allocation value, IOPS optimization control value and space occupation warning point of each node. The bandwidth dynamic allocation value is obtained by adjusting the initial bandwidth range in combination with the real-time load, which can more reasonably allocate bandwidth resources; the IOPS optimization control value is obtained by optimizing the IOPS capacity value, aiming to improve the IOPS processing efficiency of the node; the space occupation warning point is determined according to the latest storage space usage and preset strategy, which is used to timely remind the node of the tight situation of storage space.

[0079] In the process of generating the data block allocation queue, the data blocking module needs to continuously interact with the storage node state database to ensure that the obtained initial state set and dynamic load coefficient set are the latest, and at the same time, the results of dynamic adjustment are accurately reflected in the data block allocation queue.

[0080] By real-time acquisition and adjustment of the state data of the storage node, the allocation queue generated by the data blocking module can closely fit the actual storage environment. For example, when the network load of a certain storage node suddenly increases, causing the available bandwidth to decrease, the module will timely adjust the bandwidth dynamic allocation value of the node, reduce the bandwidth allocation to the node, and avoid affecting the data storage speed due to insufficient bandwidth; when the storage space of a certain node is about to reach the threshold, the module will adjust its space occupation warning point and reflect it in the data block allocation queue, so as to preferentially select other nodes with more sufficient storage space in subsequent storage operations.

[0081] This way of generating a data block allocation queue based on real-time state data enables the system to dynamically adapt to changes in the state of the storage node, avoiding problems such as uneven node load and resource waste that may occur when using fixed parameters for allocation, thereby improving the efficiency and rationality of data storage and laying an important foundation for improving the speed of server data storage. After the data block allocation queue is generated, it will be passed to the parallel optimization module for further generation of the optimal storage node combination and parallel storage scheduling strategy, ensuring that data can be stored in the most efficient way.

[0082] Embodiment 3:

[0083] When the parallel optimization module generates the parallel storage scheduling strategy based on the data block allocation queue, it needs to determine the adjustment priority queue through pattern matching with the preset storage scheduling rule library, and then generate an adaptive strategy containing node selection conditions, transmission rules and fault switching mechanisms. The specific implementation process is as follows:

[0084] After the data block allocation queue is generated, the parallel optimization module first performs pattern matching between it and the preset storage scheduling rule library. The storage scheduling rule library contains three types of core scheduling rule items. The node selection rule sets the screening conditions for the nodes to adapt to the data blocks based on the bandwidth dynamic allocation value, IOPS optimization control value, space occupation warning point and other parameters of the storage nodes, such as allowing only nodes with a bandwidth dynamic allocation value ≥ 500 Mbps to undertake large-capacity data blocks. The transmission rate adaptation rule sets the rate threshold and adjustment logic for data transmission according to the data block size and network link state, such as setting the upper limit of the transmission rate of small data blocks to 200 MB / s and the upper limit of the transmission rate of large data blocks to 500 MB / s to avoid link congestion caused by excessively high rate. The fault switching rule specifies the switching trigger conditions and data migration path of the standby node when the storage node has a response delay exceeding the threshold, IOPS fluctuation anomaly and other faults, such as starting switching when the node response delay exceeds the threshold for 3 consecutive times and preferentially migrating to a low-load standby node in the same machine room. The storage scheduling rule library pre-stores multiple rule templates for different storage scenarios, and each rule template corresponds to a different scheduling strategy combination. The data block allocation queue contains information such as the bandwidth dynamic allocation value, IOPS optimization control value and space occupation warning point of each node, which constitutes the input parameters for pattern matching.

[0085] In the pattern matching process, the module compares each parameter in the data block allocation queue with the trigger conditions of each rule template in the rule library. For example, if the bandwidth dynamic allocation value of a node in the data block allocation queue is lower than the preset threshold and the space occupation warning point reaches the high level, the rule template in the rule library for the "low bandwidth high space occupation" scenario is matched. The matching process uses a multi-dimensional feature extraction method to analyze the numerical range, change trend and mutual correlation of parameters such as bandwidth, IOPS and space occupation to determine the rule template that best matches the current allocation queue state.

[0086] Through pattern matching, the parallel optimization module determines the scheduling rule adjustment priority queue. The queue is sorted from high to low according to the matching degree of the rule template and the data block allocation queue, and the higher the matching degree of the rule template, the higher the priority of the rule template in the queue. The determination of the priority is based on a matching degree calculation model, and the formula of the model is:

[0087]

[0088] wherein, represents the matching degree of the rule template and the data block allocation queue. The number of parameter dimensions participating in matching, for example, in the present embodiment It is preferable to take 3, corresponding to the bandwidth, IOPS, and space occupation dimensions. The weight coefficient of the first parameter dimension, which takes a value in the range of , and satisfies The weight coefficient is pre-set according to the degree of influence of different parameters on storage performance, for example, the weight of the bandwidth parameter can be 0.4, the weight of the IOPS parameter can be 0.35, and the weight of the space occupation parameter can be 0.25. The actual value of the first parameter in the data block allocation queue, such as the bandwidth dynamic allocation value, the IOPS optimization control value, etc. The target threshold of the first parameter in the rule template. The matching function of the first parameter dimension, used to calculate the degree of compliance of the actual parameter value with the target threshold. The function can take the form of a piecewise function, which returns a higher value when within a preset error range of , and a lower value otherwise.

[0089] The priority queue is adjusted based on the scheduling rules, and the parallel optimization module generates an adaptive scheduling strategy. The strategy first defines the node selection conditions, and sets the filtering criteria for the storage nodes according to the requirements of the high-priority rule templates in the priority queue. For example, if the high-priority rule corresponds to a "high bandwidth and low load" scenario, the node selection conditions may include that the bandwidth dynamic allocation value is greater than a certain threshold, and the IOPS optimization control value is lower than 70% of the maximum capacity of the current node. The setting of the node selection conditions needs to consider multiple parameter dimensions to ensure that the selected nodes can meet the storage performance requirements of the data blocks.

[0090] In terms of data block transmission rules, the adaptive scheduling strategy determines the transmission order, block size adjustment method, etc. according to the rules in the priority queue. For high-priority data blocks, a priority transmission strategy can be used, and the transmission block size of the data blocks can be dynamically adjusted according to the bandwidth of the target storage node to fully utilize the node bandwidth resources. For example, when the target node bandwidth is high, the transmission block size can be appropriately increased to reduce the transmission frequency; when the bandwidth is low, the block size can be reduced to avoid transmission congestion caused by too large block size.

[0091] ​The failover mechanism is an important part of the adaptive scheduling strategy. The strategy sets the switching logic when a storage node fails according to the fault handling rules in the priority queue. For example, when a node has an IOPS anomaly or a sudden bandwidth drop during data transmission, the system will automatically switch to a backup node according to the failover mechanism and recalculate the allocation of data blocks. The selection of the backup node is based on the node selection conditions, and the node with the highest matching degree is selected from the current available nodes to ensure the continuity and performance stability of the data storage process after the failover.

[0092] When generating the adaptive scheduling strategy, the module also considers the association between data blocks and the storage order requirements. For data blocks with dependency relationships, the strategy ensures that their storage order conforms to the application logic, avoiding data from being unable to be normally used due to incorrect storage order. At the same time, the strategy coordinates the parallel storage of multiple data blocks, reasonably allocates node resources, reduces resource competition between nodes, and improves overall storage efficiency.

[0093] During the entire generation process, the parallel optimization module continuously optimizes the scheduling rules, priority queue, and adaptive scheduling strategy based on the real-time changes in the data block allocation queue and the updates of the storage node state. For example, when the load state of a storage node changes, the module recalculates its matching degree with each rule template, adjusts the priority queue, and accordingly updates the node selection conditions and transmission rules in the adaptive scheduling strategy, ensuring that the strategy is always adapted to the current storage environment, thereby continuously optimizing the data storage speed of the server.

[0094] Embodiment 4:

[0095] The performance monitoring module and the strategy matching module perform the key task of real-time monitoring of storage performance and matching and adjustment of strategies in the system. Specifically, the performance monitoring module obtains real-time storage performance data streams through the performance probes deployed on the storage nodes, filters abnormal data and sends it to the strategy matching module, and the strategy matching module performs multi-dimensional analysis and matching on the abnormal data to generate a matching degree index to determine the target adjustment rule. The following will elaborate its implementation mode in combination with specific examples.

[0096] Taking the storage of user order data in a server cluster of an e-commerce platform as an example, the performance probe of the performance monitoring module starts real-time monitoring when the data block transmission process is started. The performance probe is deployed in the operating system kernel layer or the storage service process of each storage node to collect data in a non-intrusive manner. For example, when node A transmits an order data block, the probe continuously obtains the response delay value, data transmission rate fluctuation, and IOPS stability index of the node. Assuming that at a certain time, the performance probe of node A records that the response delay value suddenly rises from an average of 5 ms to 28 ms, the data transmission rate suddenly drops from 120 MB / s to 45 MB / s, and the IOPS stability index shows that the current IOPS value fluctuates from 800 times to 300 times within 10 seconds and then rises to 700 times again, this set of data constitutes a real-time storage performance data stream.

[0097] The performance monitoring module has a preset qualified performance range, which is determined based on historical storage data and system performance baseline. For example, the qualified range of response delay is set to 0-10 ms, the qualified range of data transmission rate is 80-150 MB / s, and the qualified fluctuation range of IOPS stability index is ±10%. When the real-time monitored response delay of node A is 28 ms, which exceeds the threshold of 10 ms, the transmission rate is 45 MB / s, which is lower than the lower limit of 80 MB / s, and the IOPS fluctuation amplitude exceeds ±10%, the module automatically activates the abnormal marker, extracts the complete performance data of the abnormal period (such as 14:30:20-14:30:30), including the delay value, rate fluctuation curve, and IOPS instantaneous value collected every 500 ms, as effective monitoring data and sends it to the strategy matching module.

[0098] After receiving the abnormal performance data, the strategy matching module starts a multi-dimensional matching process. Taking the response delay increment, transmission rate change gradient, and IOPS fluctuation amplitude in the abnormal data as the core analysis objects. For example, the response delay increment of node A is 28 ms-5 ms=23 ms, the transmission rate change gradient is (45 MB / s-120 MB / s) / 10 s=-7.5 MB / s·s⁻¹, and the IOPS fluctuation amplitude is (700-300) / 800×100%=50%. The module simultaneously calls each scheduling rule item in the parallel storage scheduling strategy, and each rule item corresponds to a set of processing parameters for different performance abnormalities.

[0099] Suppose there are rule items R1 and R2 in the scheduling strategy: rule item R1 is for the "high latency low rate" scenario, the bandwidth dynamic allocation value is set to 80% of the node available bandwidth, the IOPS optimization control value is 60% of the node maximum IOPS, and the space occupation warning point is 75% of the storage space; rule item R2 is for the "IOPS dramatic fluctuation" scenario, the bandwidth dynamic allocation value is 70%, the IOPS optimization control value is 50%, and the space occupation warning point is 80%. The strategy matching module calculates the difference between the abnormal data and each rule item.

[0100] For rule item R1, the difference between the response delay increment 23 ms and the delay processing threshold corresponding to the rule item (preset as 20 ms) is 3 ms, the difference between the transmission rate change gradient -7.5 MB / s·s⁻¹ and the rate change threshold in the rule item (-5 MB / s·s⁻¹) is -2.5 MB / s·s⁻¹, and the difference between the IOPS fluctuation amplitude 50% and the IOPS fluctuation threshold in the rule item (40%) is 10%. For rule item R2, the difference between the IOPS fluctuation amplitude 50% and the IOPS fluctuation threshold of the rule item (60%) is -10%, the difference between the response delay increment 23 ms and the delay threshold of the rule item (25 ms) is -2 ms, and the difference between the transmission rate change gradient is -2.5 MB / s·s⁻¹.

[0101] After the difference calculation, the module generates a matching degree index in a multi-dimensional weighting manner. The weight allocation is based on the influence degree of each parameter on the storage performance, for example, the delay increment weight is set to 0.4, the rate change gradient weight is 0.3, and the IOPS fluctuation amplitude weight is 0.3. For rule item R1, its matching degree is calculated as follows: the delay difference 3 ms corresponds to a matching degree score (using reverse mapping, the smaller the difference, the higher the score, for example, 0-5 ms corresponds to 90-100 points, and 5-10 ms corresponds to 80-90 points) of about 85 points, the rate change gradient difference -2.5 MB / s·s⁻¹ corresponds to a score of about 80 points, the IOPS fluctuation amplitude difference 10% corresponds to a score of about 70 points, and the comprehensive matching degree is 85x0.4+80x0.3+70x0.3=80 points. The delay difference -2 ms of rule item R2 corresponds to a score of about 95 points, the rate change gradient difference corresponds to a score of 80 points, and the IOPS fluctuation amplitude difference -10% corresponds to a score of about 85 points. The comprehensive matching degree is 95x0.4+80x0.3+85x0.3=87.5 points.

[0102] Through the above calculation, the matching degree of rule item R2 is 87.5 points, which is higher than that of R1, i.e. 80 points, and the policy matching module determines R2 as the scheduling rule with the highest matching degree at present. If there are multiple rule items, the module will establish a matching degree ranking list and select the first rule as the target adjustment rule. If the first matching degree is lower than a preset trigger threshold (such as 70 points), a backup rule set is called to re-match. In this example, since 87.5 points is higher than the threshold, R2 is selected as the target adjustment rule, and the scheduling parameters (such as bandwidth allocation 70%, IOPS control 50%) contained in R2 will be used for subsequent storage path correction.

[0103] During the whole process, the performance monitoring module continuously samples the performance data stream of all storage nodes in real time, and the sampling frequency can be dynamically adjusted according to the system load (such as 10 ms / time under high load and 100 ms / time under low load), to ensure that the abnormality is captured in time. The policy matching module maintains a rule item cache pool to cache recently frequently matched rules to speed up the matching process. For example, during the e-commerce promotion period, IOPS fluctuation abnormalities frequently occur in the storage nodes, and rule item R2 will be cached. When the same type of abnormality occurs subsequently, the cached rule can be directly called to perform quick matching, reducing the calculation time.

[0104] In addition, the system supports dynamic updating of rule items. When a rule item shows a continuously low matching degree in multiple matches, and after being confirmed by manual review, the parameter threshold of the rule item can be updated through the management interface to adapt to the changes of the storage environment. For example, if the IOPS processing capacity of node A is improved due to hardware upgrade, the IOPS fluctuation threshold in the original rule item can be adjusted accordingly to avoid the deviation of the matching degree calculation caused by the non-updated threshold.

[0105] Through the linkage mechanism of performance monitoring and policy matching, the system can quickly locate the corresponding adjustment strategy for different types of storage performance abnormalities, provide accurate adjustment basis for the storage execution module, and thus realize dynamic optimization of the server data storage speed. In the above example, for the IOPS fluctuation abnormality of node A, by matching rule item R2 and performing corresponding bandwidth and IOPS control adjustment, the node load pressure can be effectively relieved, and the data transmission efficiency can be restored, which reflects the pertinence and effectiveness of the implementation in practical application.

[0106] Embodiment 5:

[0107] The policy matching module in the system plays a key role in selecting the target adjustment rule and the load self-learning module in optimizing the scheduling strategy. The implementation thereof will be described in detail below in combination with specific examples.

[0108] Taking a server cluster of a certain financial institution storing transaction data as an example, when the storage node B is transmitting a large transaction data block, the performance monitoring module detects that the response delay of the node B suddenly increases, and the strategy matching module starts the matching process. Assuming that at this time the real-time performance data shows that the response delay increment is 25 ms, the transmission rate change gradient is -8 MB / s-s-1, and the IOPS fluctuation amplitude is 45%. The strategy matching module matches these abnormal data with each scheduling rule item in multiple dimensions, and calculates that the matching degree of rule item R3 is the highest, which is 88 points, higher than the preset adjustment trigger threshold of 70 points, so R3 is determined as the target adjustment rule. The scheduling adjustment protocol corresponding to the rule item R3 is: reducing the bandwidth dynamic allocation value of node B from the current 75% to 60%, adjusting the IOPS optimization control value from 500 times to 350 times, and triggering the load balancing mechanism of the standby node C.

[0109] After receiving the target adjustment rule, the storage execution module calls the preset scheduling adjustment protocol in the storage node state database to drive the storage nodes B and C to perform the data storage path correction operation. After completing this storage path correction, the load self-learning module starts to record the data block storage feedback data after this operation. Specifically, the feedback data includes actual transmission completion time, node load balancing degree, and data integrity check result, and the like. For example, the actual transmission completion time of this large transaction data block is 18 seconds, and the expected transmission time before adjustment is 25 seconds; the load balancing degree of node B after adjustment decreases from the original 85% to 60%, and the load balancing degree of node C increases from 30% to 45%; the data integrity check result shows that all data blocks are complete and correct, and there is no packet loss or damage phenomenon.

[0110] The load self-learning module performs reverse verification on these feedback data and the parallel storage scheduling strategy. The process of reverse verification is to compare and analyze the actual feedback data with the expected performance target in the strategy to determine the effectiveness of the strategy and the direction that needs to be optimized. In this example, the expected transmission time is 25 seconds, and the actual transmission time is 18 seconds, indicating that the transmission efficiency has improved; the load balancing degree changes of nodes B and C show that the load balancing mechanism has played a certain role. Based on the deviation degree of the feedback data from the expected performance target, the load self-learning module calculates the bandwidth optimization factor, the IOPS adjustment factor, and the space management weight.

[0111] Assuming that the deviation between the expected transmission time and the actual transmission time is 7 seconds, the bandwidth optimization factor is calculated based on the deviation. The calculation of the bandwidth optimization factor takes into account the impact of transmission time deviation on bandwidth allocation. If the actual transmission time is less than the expected, it may mean that the current bandwidth allocation is reasonable or can be adjusted appropriately to optimize the resource allocation of other nodes. Similarly, the IOPS adjustment factor is calculated based on the change in the load balancing degree of the nodes. The load balancing degree of node B has decreased by 25%, and that of node C has increased by 15%. This indicates that the adjustment of IOPS has an impact on load balancing, and the IOPS adjustment factor needs to be calculated accordingly. The space management weight is determined based on the usage of the storage space and other relevant information in the feedback data.

[0112] After calculating the factors and weights, the load self-learning module uses the exponential weighted average algorithm to dynamically smooth the historical correction factors to generate a new set of dynamic load coefficients. The historical correction factors refer to the adjustment factors generated after each previous storage path correction. The exponential weighted average algorithm allows the newly generated dynamic load coefficient set to consider both current feedback data and historical experience, making the adjustment more smooth and reasonable. For example, the historical bandwidth optimization factor values are 0.8, 0.75, and 0.82. Combined with the current calculated bandwidth optimization factor of 0.9, the new bandwidth allocation weight may be adjusted to 0.78 after calculation by the exponential weighted average algorithm.

[0113] After generating the new set of dynamic load coefficients, it is updated to the dynamic load coefficient set in the storage node state database. The dynamic load coefficient set in the storage node state database includes bandwidth allocation weight, IOPS priority coefficient, and space occupation adjustment factor, etc. After updating, these parameters will be used for subsequent data block allocation queue generation and scheduling strategy optimization.

[0114] In the subsequent storage process, when similar large transaction data is processed again, the data block allocation module will generate a data block allocation queue based on the updated dynamic load coefficient set. For example, when generating the allocation queue, the bandwidth allocation weight of node B may be adjusted based on the new dynamic load coefficient, making the allocation of bandwidth resources more reasonable and avoiding the occurrence of high load again. The parallel optimization module also refers to the updated dynamic load coefficient when generating the parallel storage scheduling strategy, thereby generating a strategy that is more in line with the actual storage environment.

[0115] In addition, the load self-learning module continuously records the feedback data after each storage path correction and repeatedly performs the above process, allowing the system to continuously optimize the scheduling strategy based on the actual storage situation. For example, after processing large transaction data multiple times, the system continuously adjusts the dynamic load coefficient through self-learning, making the load of the storage nodes more balanced and the data transmission speed more stable when processing this type of data.

[0116] In another example, if the actual transmission completion time is 30 seconds after the storage path is corrected, which exceeds the expected 25 seconds, the feedback data shows that the load balancing degree of node D is still as high as 90%, while the load balancing degrees of other nodes are lower. At this time, the load self-learning module will find that the strategy is insufficient during the reverse verification, and the bandwidth optimization factor and IOPS adjustment factor calculated will reflect that the bandwidth allocation and IOPS control of node D need to be re-adjusted. After processing the historical correction factors by the exponential weighted average algorithm, the new dynamic load coefficient set generated will adjust the related parameters of node D to reduce its load and improve the overall storage efficiency.

[0117] Through the continuous learning and optimization of the load self-learning module, the system can adapt to different storage scenarios and data types, and gradually improve the server data storage speed and stability.

[0118] It should be noted that, in this text, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0119] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A processing system for improving server data storage speed, characterized in that, include: The data segmentation module is used to receive the data stream to be stored, divide the data stream into multiple data blocks according to a predefined dynamic segmentation strategy, and generate a data block allocation queue based on the storage node status database. The parallel optimization module, based on the data block allocation queue, processes and generates the optimal combination of storage nodes for each data block and generates a parallel storage scheduling strategy. The performance monitoring module is used to obtain real-time storage performance data streams through performance probes deployed on storage nodes, filter out abnormal performance data that meets the dynamic adjustment range, and send them to the policy matching module. The strategy matching module is used to perform multi-dimensional matching between abnormal performance data and each scheduling rule item in the parallel storage scheduling strategy, generate the matching degree between real-time performance data and each scheduling rule item, and select the scheduling rule corresponding to the highest matching degree as the target adjustment rule. The storage execution module is used to receive the target adjustment rules and call the preset scheduling adjustment protocol in the storage node status database to drive the storage node to perform data storage path correction operations; It also includes a load self-learning module, specifically including: Record the data block storage feedback data after each storage path correction, including the actual transmission completion time, node load balancing, and data integrity verification results; The feedback data is used to verify the parallel storage scheduling strategy in reverse, and a scheduling correction factor is generated and updated to the dynamic load coefficient set in the storage node status database. The generation of the scheduling correction factor specifically includes: Based on the degree of deviation between the feedback data and the expected performance target, calculate the bandwidth optimization factor, IOPS adjustment factor, and space management weight; An exponentially weighted average algorithm is used to dynamically smooth the historical correction factors, generating a new set of dynamic load coefficients.

2. The processing system for improving server data storage speed according to claim 1, characterized in that: The process of dividing the data stream into multiple data blocks according to a predefined dynamic block division strategy is as follows: The unique identifier of the data stream is identified. If it is a new data stream type, the basic block parameters are collected, including the initial block size range, priority weight factor and redundancy configuration baseline value. Based on the initial data, block feature nodes are established and policy calibration is performed to generate a data block allocation queue. If it is a historical data stream type, extract the historical block dataset and storage performance change curve of that type. The historical block dataset includes block size adjustment frequency, node load balancing records and transmission latency fluctuation values, and is marked as the initial parameter set for the current block processing.

3. The processing system for improving server data storage speed according to claim 1, characterized in that: The generation of the data block allocation queue based on the storage node status database specifically includes: Extract the initial state set and dynamic load coefficient set of each storage node from the storage node state database. The initial state set includes: the available bandwidth range of the node, the current IOPS capacity value, and the remaining storage space threshold. The dynamic load factor set includes bandwidth allocation weight, IOPS priority factor, and space occupancy adjustment factor; Based on real-time storage node status data, the initial state set of each storage node is dynamically adjusted, and the adjusted state set is recorded as the data block allocation queue. The data block allocation queue includes the dynamic bandwidth allocation value, IOPS optimization control value, and space occupancy warning point for each node.

4. The processing system for improving server data storage speed according to claim 3, characterized in that: The process of generating the optimal combination of storage nodes for each data block and generating a parallel storage scheduling strategy based on the data block allocation queue specifically includes: The data block allocation queue is matched with a preset storage scheduling rule base to determine the scheduling rules and adjust the priority queue. An adaptive scheduling strategy is generated based on adjusting the priority queue, which includes node selection conditions, data block transmission rules, and failover mechanisms.

5. The processing system for improving server data storage speed according to claim 4, characterized in that: The process of obtaining real-time storage performance data streams through performance probes deployed on storage nodes specifically includes: Monitor the real-time storage performance data stream during the data block transmission process, including node response latency, data transmission rate fluctuations, and IOPS stability metrics; When the real-time storage performance data stream exceeds the preset acceptable performance range, an anomaly marker is activated and the performance data of the abnormal period is extracted as valid monitoring data.

6. The processing system for improving server data storage speed according to claim 4, characterized in that: The matching degree between the generated real-time performance data and each scheduling rule item specifically includes: The response latency increment, transmission rate change gradient, and IOPS fluctuation amplitude in the abnormal performance data are analyzed, and the difference is calculated with the bandwidth dynamic allocation value, IOPS optimization control value, and space occupancy warning point of each scheduling rule item, respectively. Based on the difference calculation results, a matching index is generated between real-time performance data and each scheduling rule item.

7. The processing system for improving server data storage speed according to claim 1, characterized in that: The step of selecting the scheduling rule corresponding to the highest matching degree as the target adjustment rule specifically includes: Establish a sorted list of matching scores for each scheduling rule item, and select the rule item with the highest matching score in the list; If the first matching degree is lower than the preset adjustment trigger threshold, the backup scheduling rule set is called and the matching degree is recalculated.

8. The processing system for improving server data storage speed according to claim 6, characterized in that: The specific process for generating the matching degree index between real-time performance data and each scheduling rule item based on the difference calculation results is as follows: A multi-dimensional matching algorithm is used to standardize the bandwidth difference, IOPS deviation, and space usage difference, generating a scheduling rule matching degree value ranging from 0 to 100; The closer the matching degree value is to 100, the stronger the adaptability between real-time performance data and scheduling rules.

Images