Multi-Pool storage resource adaptive writing method and device, equipment and medium

Abstract

The invention relates to the technical field of data storage, can be applied to business scenes such as medical health and financial science and technology, and discloses a multi-Pool storage resource adaptive writing method which comprises the following steps: monitoring available storage space, read-write input / output rate, average response time and data access frequency of each Pool; dynamically adjusting the current write-in weight of the Pool according to the monitoring result; converting the current write-in weights of all Pool into probability values, and constructing a discrete probability distribution table; selecting a target Pool through a pseudo-random number generator according to the probability distribution table; and writing the to-be-stored data into the target Pool. According to the method, the running state of each Pool is continuously monitored, and the writing weight is adjusted in real time, so that the problem of resource allocation rigidity under a fixed weight strategy is avoided; in combination with a discrete probability distribution table constructed based on normalized weight and a pseudo-random selection mechanism, selection of the target Pool has certain load balancing elasticity, and data can be more effectively guided to be written into the Pool with light load and excellent performance.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and particularly to a method, device, equipment and storage medium for adapting and writing multi-Pool storage resources. Background Art

[0002] In the technical field of multi-Pool data storage systems, traditional data writing strategies mainly rely on static weight setting mechanisms. That is, at the initial stage of system deployment, the bearing capacity of each Pool is estimated based on its hardware configuration, capacity resources or service roles, so as to assign a fixed writing weight to each Pool. Such mechanisms have a certain adaptability in the scenarios of initial system deployment or small load changes. However, with the increase in business complexity and the explosion of data volume, the load status, response ability and data access patterns of each Pool will change frequently during the operation of the system, but the originally set static weights are difficult to adjust in real time, resulting in unbalanced resource scheduling. For example, when the performance of a certain Pool decreases due to long-term high-load writing, the system still imports new data continuously according to the original weight, resulting in an increase in response latency and even causing writing congestion, affecting the availability and throughput efficiency of the entire system.

[0003] In the field of medical and health services, medical information systems often need to process a large number of data writing requests during periods such as concentrated patient visits, synchronous clinical diagnoses or batch uploads of examination images. Different business modules (such as outpatient records, test reports, image data, etc.) will generate access peaks in a short period of time, and the corresponding underlying storage Pools will also be under different degrees of writing pressure. However, the traditional static writing strategy lacks the ability to adjust such periodic writing peaks, which easily causes delays in writing critical data and may even affect the accuracy and timeliness of data calls in subsequent diagnosis and treatment systems, thus interfering with the overall medical service process.

[0004] In the field of fintech services, data writing has characteristics of high timeliness and strong business fluctuations. For example, during peak trading matching periods, risk warning sounding periods or time windows when batch settlement tasks are executed intensively, the data inflow volume associated with specific business modules will increase significantly, forming high-intensity data writing hotspots. If the system uses fixed writing weights and lacks the ability to dynamically perceive and adjust real-time data access hotspots, it will cause some hot Pools to be overwritten and their performance to drop sharply, while other idle Pools are not effectively utilized, resulting in a decrease in resource utilization efficiency and may have an adverse impact on the writing consistency and transaction continuity of critical financial data. Summary of the Invention

[0005] The main objective of the present invention is to provide a method, device, equipment, and storage medium for adaptively writing multi-Pool storage resources, aiming to solve the technical problem in the prior art that the writing weight cannot be adjusted in real time according to the change of the Pool operation state, resulting in low resource utilization rate and decreased writing performance.

[0006] To achieve the above objective, the present invention provides a method for adaptively writing multi-Pool storage resources, including:

[0007] Monitoring the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool;

[0008] When it is monitored that the available storage space of a Pool is lower than the preset space threshold, the current writing weight of the Pool with the available storage space lower than the preset space threshold is reduced;

[0009] When it is monitored that the read / write input / output rate of a Pool is higher than the preset rate benchmark, the current writing weight of the Pool with the read / write input / output rate higher than the preset rate benchmark is increased;

[0010] When it is monitored that the data access frequency of a Pool exceeds the preset frequency threshold, the current writing weight of the Pool with the data access frequency exceeding the preset frequency threshold is increased;

[0011] When it is monitored that the average response time of a Pool exceeds the preset time threshold, the current writing weight of the Pool with the average response time exceeding the preset time threshold is reduced;

[0012] If a Pool is in an abnormal state, the current writing weight of the Pool in the abnormal state is set to zero;

[0013] Converting the current writing weight of each Pool into a probability value, and constructing a discrete probability distribution table based on the probability value;

[0014] Determining the target Pool through a pseudo-random number generator according to the discrete probability distribution table;

[0015] Writing the data to be stored into the target Pool.

[0016] Furthermore, to achieve the above objective, the present invention provides a device for adaptively writing multi-Pool storage resources, including:

[0017] A monitoring module, configured to monitor the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool;

[0018] A space regulation module, configured to reduce the current write weight of a Pool whose available storage space is lower than a preset space threshold when it is detected that there is a Pool with available storage space lower than the preset space threshold;

[0019] A rate regulation module, configured to increase the current write weight of a Pool whose read / write input / output rate is higher than a preset rate benchmark when it is detected that there is a Pool with a read / write input / output rate higher than the preset rate benchmark;

[0020] A frequency regulation module, configured to increase the current write weight of a Pool whose data access frequency exceeds a preset frequency threshold when it is detected that there is a Pool with a data access frequency exceeding the preset frequency threshold;

[0021] A response regulation module, configured to reduce the current write weight of a Pool whose average response time exceeds a preset time threshold when it is detected that there is a Pool with an average response time exceeding the preset time threshold;

[0022] An exception handling module, configured to set the current write weight of a Pool in an abnormal state to zero if there is a Pool in an abnormal state;

[0023] A probability construction module, configured to convert the current write weight of each Pool into a probability value and construct a discrete probability distribution table based on the probability value;

[0024] A target selection module, configured to determine a target Pool according to the discrete probability distribution table through a pseudo-random number generator;

[0025] A write execution module, configured to write data to be stored into the target Pool.

[0026] Furthermore, to achieve the above object, the present invention further provides a computer device, which includes a memory, a processor, and a multi-Pool storage resource adaptation write program stored in the memory and executable on the processor. When the multi-Pool storage resource adaptation write program is executed by the processor, the steps of the multi-Pool storage resource adaptation write method as described above are implemented.

[0027] Furthermore, to achieve the above object, the present invention further provides a computer-readable storage medium, on which a multi-Pool storage resource adaptation write program is stored. When the multi-Pool storage resource adaptation write program is executed by a processor, the steps of the multi-Pool storage resource adaptation write method as described above are implemented.

[0028] Beneficial Effects: The present invention relates to the field of data storage technology and can be applied to business scenarios such as medical health and fintech. It discloses a multi-Pool storage resource adaptation writing method, including: monitoring the available storage space, read-write input-output rate, average response time, and data access frequency of each Pool; dynamically adjusting the current writing weight of the Pool according to the monitoring results; converting the current writing weights of all Pools into probability values to construct a discrete probability distribution table; selecting a target Pool according to the probability distribution table through a pseudo-random number generator; and writing the data to be stored into the target Pool. By continuously monitoring the operating status of each Pool and adjusting the writing weight in real time, the present invention avoids the problem of rigid resource allocation under the fixed weight strategy; combined with the discrete probability distribution table constructed based on the normalized weight and the pseudo-random selection mechanism, the selection of the target Pool has a certain load balancing flexibility, and can more effectively guide the data writing to the Pool with lighter load and better performance; at the same time, through the abnormal state detection mechanism, the unavailable or performance-abnormal Pools are actively isolated, further enhancing the robustness and writing fault tolerance of the system, thereby significantly improving the writing efficiency, system stability, and resource utilization rate in the high-frequency trading environment in the fintech business field and the high-concurrency data recording scenario in the medical health business field. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The following will further illustrate the present invention with reference to the drawings. In the drawings:

[0030] Figure 1 is a schematic diagram of an application environment of the multi-Pool storage resource adaptation writing method in an embodiment of the present invention;

[0031] Figure 2 is a schematic flowchart of an embodiment of the multi-Pool storage resource adaptation writing method of the present invention;

[0032] Figure 3 is a schematic diagram of the functional modules of a preferred embodiment of the multi-Pool storage resource adaptation writing device of the present invention;

[0033] Figure 4 is a schematic diagram of the structure of a computer device in an embodiment of the present invention;

[0034] Figure 5 is another schematic diagram of the structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] The multi-Pool storage resource adaptation writing method provided by the embodiments of the present invention can be applied in such as Figure 1In the application environment, the client communicates with the server through the network. The server can monitor the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool through the client; dynamically adjust the current write weight of the Pool according to the monitoring results; convert the current write weights of all Pools into probability values to construct a discrete probability distribution table; select a target Pool according to the probability distribution table through a pseudo-random number generator; and write the data to be stored into the target Pool. By continuously monitoring the running status of each Pool and adjusting the write weight in real time, the present invention avoids the problem of rigid resource allocation under the fixed weight strategy; combines the discrete probability distribution table constructed based on the normalized weight and the pseudo-random selection mechanism, so that the selection of the target Pool has a certain load balancing flexibility and can more effectively guide the data to be written into the Pool with lighter load and better performance; at the same time, through the abnormal state detection mechanism, the unavailable or abnormally performing Pools are actively isolated, further enhancing the robustness and write fault tolerance of the system, thereby significantly improving the write efficiency, system stability, and resource utilization rate in the high-frequency trading environment in the fintech business field and the high-concurrency data recording scenario in the healthcare business field. Among them, the client can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail below through specific embodiments.

[0037] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of an embodiment of the multi-Pool storage resource adaptation writing method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than here.

[0038] As Figure 2 shown, the multi-Pool storage resource adaptation writing method proposed by the present invention includes the following steps:

[0039] S10, monitor the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool;

[0040] In this embodiment, in the multi-Pool data storage system, each Pool is composed of a group of physical or logical storage nodes to support read / write requests for data. When the system faces high-concurrency and large-scale data write requests, resource allocation decisions need to be made according to the running status of each Pool. Therefore, collecting and real-time updating the running metrics of each Pool is the basis for realizing dynamic write optimization.

[0041] The available storage space is a direct indicator of whether the Pool can carry new data, usually sourced from the capacity information exposed by the underlying file system or storage controller. The system can obtain the ratio of the remaining capacity to the total capacity by periodically calling interfaces, such as POSIX-compatible disk status query APIs or block device driver monitoring interfaces, thereby forming a standardized numerical representation of the available space.

[0042] The read / write input / output rate refers to the number of read / write operations completed per unit time. It can be collected by the IO statistics module for the IOPS (Input / Output Operations Per Second) data of active storage nodes in each Pool within a set time window, or calculated in real time from the transaction records in the operation logs. The statistics within the sliding window help filter out short-term burst interference and reflect a more stable performance trend.

[0043] The average response time reflects the time consumption of the Pool in processing a single request, often limited by a combination of factors such as hardware performance, task queue depth, and data locality. The system can record the sending time when sending an IO request to the storage subsystem and record the end time when receiving the operation completion feedback, and calculate the interval as the single response time. Then, taking the arithmetic mean of the response times of multiple requests can obtain the current average response performance of the Pool. In distributed storage, it is often necessary to exclude the network layer latency, so corrections can be made in combination with the reference latency measured by the network layer.

[0044] The data access frequency is an important dimension for evaluating the current business pressure of each Pool, reflecting not only the data popularity but also the concentration of the business scenario. Usually, the access volume of each type of request can be extracted from user access logs, system call chains, and object storage header information. For example, the system can classify and aggregate according to the user identifier of the access source and the access data type (log data, transaction data, clinical data, etc.) to obtain the access frequency vector of each Pool per unit time.

[0045] The above metrics can be refreshed at a millisecond or second level frequency and stored in an in-memory database or cache structure to support reading at any time and quick decision-making. They are not isolated but constitute four dimensions representing the comprehensive load status of each Pool. Subsequent actions such as weight adjustment and target selection are based on this. The metrics collected by the technical solution are not only horizontally comparable (among various Pools) but also vertically traceable (over different time periods), thus constructing a stable resource evaluation basis.

[0046] In actual deployment, the system can deploy lightweight metric collection plugins for each Pool based on the resource monitoring agent. The plugins run on the nodes where the Pools are located or in the access layer, maintain regular heartbeats with the master control module, and report the latest metrics. The available storage space can obtain the remaining block count or inode usage rate of the current disk through the / dev or / proc directory; the read / write rate can build a sliding window array based on dstat, iostat, or a custom IO monitoring module, and record the increment of read / write times for each sample; the average response time can insert a timestamp hook when calling the block device driver function to obtain the request lifecycle and eliminate outliers; the data access frequency analyzes the Nginx logs, HDFS audit logs, or S3 request logs of object storage, extracts the access source and resource identifier, and aggregates the traffic hotspots for each Pool.

[0047] In terms of the deployment solution, the system can choose a centralized model where a global controller collects the metrics of all Pools, or adopt an edge collaboration model where preliminary evaluation is performed in the local control units of each Pool and the summaries are uploaded. For scenarios with higher real-time requirements, a memory database (such as Redis) can be combined as an intermediate cache to asynchronously write the sampled data into the main database to ensure the collection efficiency.

[0048] In addition, the sampling frequency, window size, and metric accuracy can be adjusted for different scenarios. In high-concurrency write scenarios, the response time and write rate require a higher sampling density; in long-term analysis, the space usage rate and access frequency can be analyzed for trends over a longer period. The above parameters can all be dynamically adjusted by the policy module and persistently stored in the policy configuration center to support runtime adjustment.

[0049] Example illustration: In the fintech business domain, the transaction matching system needs to write high-frequency data into multiple log pools or audit pools. If a certain Pool becomes slower to respond due to the backlog of concurrent transaction records in a short period, and the traditional system still allocates write operations according to a fixed weight, it will cause latency backlogs and even data loss. After adopting this method, the system monitors the response time and IO rate of this Pool in real time, reduces its write weight in a timely manner when the performance deteriorates, and automatically guides the write traffic to other Pools with lighter loads to ensure the highly available operation of the transaction system.

[0050] In the healthcare business domain, the image storage system needs to process the CT / MRI data transmission from different departments and terminals. During specific periods (such as peak examination times), the access volume of certain Pools surges, and the traditional mechanism is difficult to identify the hotspot changes in a timely manner, easily forming an IO bottleneck. After adopting this method, the system can sense the upward trend of the access frequency and dynamically adjust the weight, so that high-frequency requests preferentially fall into the Pools with better response capabilities, thereby improving the image upload and archiving efficiency.

[0051] In the field of distributed storage system operation and maintenance, data lakes or cloud storage platforms continuously collect and evaluate the resource status of each Pool through this method, providing an intelligent load evaluation basis for operation and maintenance personnel, and supporting the adaptive resource scheduling and automatic expansion strategies of the upper-layer decision-making module, thereby enhancing the system's autonomy and resource management efficiency.

[0052] By monitoring the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool, the system can comprehensively grasp the operating status of each storage resource pool, perceive the current load and performance bottlenecks from multiple dimensions, and thus provide an objective basis for subsequent write weight adjustment and target selection. It can respond in real time to system state changes, effectively avoid resource misallocation caused by misjudgment of a single indicator, and improve the accuracy of the write path and the stability of data disk writing. The overall write performance of the system is improved, the fault tolerance ability is enhanced, and the resource utilization efficiency is optimized.

[0053] S20. When it is detected that the available storage space of a Pool is lower than the preset space threshold, the current write weight of the Pool with the available storage space lower than the preset space threshold is reduced;

[0054] In this embodiment, the available storage space of the Pool is an important basic indicator for measuring its ability to carry write data. When the available space of a certain Pool continues to decrease, it means that it is getting closer to the full storage state. Continuing to maintain the current write speed may cause data overwriting, system crashes, or trigger abnormal expansion operations. Therefore, monitoring and judging whether the available space of the Pool is lower than a safety threshold is a prerequisite for realizing adaptive storage resource allocation.

[0055] The available storage space generally refers to the remaining unused physical or logical storage capacity in the current Pool. The acquisition method can rely on the query interface exposed by the underlying operating system or storage service, such as the df command of Linux or the block device driver API, or it can also be sampled through the RESTful management interface provided by the storage system. The available space can be expressed in bytes, blocks, or percentages, and percentages are more suitable for horizontal comparison between Pools.

[0056] The preset space threshold is a boundary value used to judge whether a certain Pool is in a high occupancy state. This threshold can be set comprehensively according to business tolerance, historical write rate, alarm strategy, etc. For example, it can be set to 20%, that is, when the available space is less than 20% of the total capacity, it is regarded as "insufficient space". This threshold can also be adjusted dynamically. For example, the bearable time can be predicted based on the write rate in the past week, and thus the threshold standard can be dynamically increased or decreased.

[0057] Reducing the current write weight means reducing the probability weight of the Pool being selected as the target during the write target selection process. The expression form of the write weight is usually a normalized value, for example, floating between 0 and 1. The weight can be reduced by setting a stepped decreasing ratio, an interpolation algorithm, or a linear fallback model, so that the probability of the Pool being selected when participating in the write decision is reduced, thereby reducing its load and avoiding further approaching the full capacity state.

[0058] This mechanism can not only prevent write failures caused by local space depletion, but also extend the stable operation time of the system, gain more time for capacity expansion or cold data migration and other strategy deployments, and has important operation and maintenance value.

[0059] The system can collect the available space of each Pool within each sampling period and compare it with a preset threshold. The collection mechanism can be completed by a monitoring agent deployed on each Pool management node. This agent periodically reads the remaining space metrics of the underlying file system, object storage, or block device, and summarizes and reports them to the central control module.

[0060] When determining whether a certain Pool is below the threshold, it can be done by comparing its available space percentage with the dynamic threshold. For example, if the remaining capacity of a certain Pool is 180GB and the total capacity is 1TB, the calculated available rate is 18%. If the threshold is set to 20%, then the "insufficient space" judgment is triggered.

[0061] Once the judgment logic is triggered, the system can adjust the write weight of the Pool. For example, it can be adjusted from the initial value of 0.25 to 0.1. This process can adopt a linear reduction strategy: weight adjustment value = initial weight × current available space percentage ÷ threshold. For example, when the available space is 15%, the threshold is 20%, and the initial weight is 0.2, then the adjusted weight = 0.2×(0.15 / 0.2) = 0.15.

[0062] After the weight is updated, this value will be used to construct a new probability distribution table and participate in the selection process of the target Pool. The whole process can be automatically executed without manual intervention and supports hot loading of policy updates.

[0063] Example illustration: In the field of fintech business, business systems with a sharp increase in data volume such as transaction data and account logs often face the situation that a certain storage node has intensive writes, resulting in rapid consumption of capacity. For example, on a certain day, due to a sudden transaction peak, the writes of a certain node increased sharply, and the remaining space of the Pool quickly approached the preset lower limit. Through this mechanism, the system can immediately reduce the write weight of the Pool and write the new transaction data to other nodes with more sufficient capacity, avoiding write failures caused by insufficient space of this node, thereby ensuring the high availability of the transaction system.

[0064] In the field of healthcare services, a hospital's imaging system writes a large number of CT and MRI images to a specific imaging storage pool within a certain period of time. If the space in this pool is insufficient, the system will promptly reduce its write weight and automatically write new images to a backup pool to ensure uninterrupted data writing, without affecting core business processes such as image uploading and diagnosis, thereby enhancing the stability of the diagnosis and treatment system and the data security protection ability.

[0065] By actively reducing the current write weight of a Pool when its available storage space is lower than the preset space threshold, the system can effectively avoid continuously writing to a Pool that is already approaching its full storage state, preventing local storage crashes or write failures. At the same time, this strategy redirects the write traffic to other Pools with more sufficient capacity, achieving dynamic and balanced allocation of resources. Overall, it enhances the fault tolerance of the storage system, the write success rate, and service stability, provides buffer space for subsequent data migration and expansion plans, reduces the frequency of manual intervention, and improves the level of automated operation and maintenance.

[0066] S30, when it is detected that the read / write input / output rate of a Pool is higher than the preset rate benchmark, then increase the current write weight of the Pool whose read / write input / output rate is higher than the preset rate benchmark;

[0067] In this embodiment, the read / write input / output rate (I / O rate) is an important dynamic performance indicator that characterizes the processing and response capabilities of a storage Pool. This indicator is usually measured by the number of read / write requests completed or the total amount of data transferred per unit time, and can be quantified using IOPS (Input / Output Operations Per Second) or throughput (MB / s). It can directly reflect the performance level of the current Pool under the actual load of the system.

[0068] The fact that the read / write input / output rate of a Pool is higher than the preset rate benchmark means that this Pool not only does not become a performance bottleneck, but instead has strong data processing capabilities and the potential to further carry write requests. This judgment is based on the preset rate benchmark, which can be obtained through historical performance analysis, device specification calibration, runtime self-learning, etc. Its value can be either a statically set performance reference lower limit or an index threshold dynamically corrected according to the system operation state.

[0069] In the system, continuously sample the number of read / write requests and the data transfer volume of each Pool, obtain stable statistical values through a sliding window calculation strategy, and then compare them with this benchmark value. Once it is confirmed that the I / O rate of a certain Pool is higher than the benchmark, it indicates that its processing ability is strong or the current load is low, and it has a higher access capacity.

[0070] Increasing the current write weight means that during the data write scheduling process, the probability weight of the Pool in write target selection is appropriately increased. Its mechanism includes boosting the value corresponding to the Pool in the normalized weight vector, and the boosting amplitude can be determined according to the ratio exceeding the benchmark rate. For example, the linear ratio method (the higher the exceeding ratio, the greater the weight boost) or the discrete level method (fixed amounts are boosted respectively for multiple threshold segments) can be used. The weight boost should be controlled by an upper limit to prevent a single Pool from becoming a new hot spot due to frequent preferential allocation.

[0071] This scheduling optimization mechanism can adjust the resource allocation strategy based on the real-time I / O capabilities, achieve the adaptive scheduling of storage resources when the system load changes dynamically, and thus improve the overall processing efficiency and write success rate.

[0072] For example, the system deploys a performance collection agent module at the control node of each Pool to regularly count its I / O operation volume. Every fixed period, such as 10 seconds, the read and write times and data volume of the Pool within a sliding time window (such as the past 60 seconds) are collected, and the average I / O rate is calculated. The calculation methods include: IOPS = number of requests / number of seconds in the time window, or throughput = total number of read and written bytes / number of seconds in the time window. The system maintains a preset rate benchmark, which can be set as a static reference value according to the storage device specifications (such as an SSD supporting 30,000 IOPS per second), or can be dynamically adjusted in combination with the system load situation. Assuming the benchmark is 1000 IOPS, if the current rate of a certain Pool is 1300 IOPS, exceeding the benchmark by 30%, the weight boost logic is triggered. The system can set the weight boost strategy as linear boost, for example, the weight boost ratio = current rate / benchmark rate. For a Pool with an original write weight of 0.25, after adjustment, it becomes 0.25×(1300 / 1000) = 0.325. This weight will be used to update the probability distribution table, thereby increasing the selection probability of this Pool in subsequent write scheduling. The weight update process can be automated, support real-time update after each sampling period, and allow the external policy control module to inject limiting conditions (such as the maximum weight shall not exceed 0.5) to avoid out-of-control situations.

[0073] By boosting the write weight of a Pool when its read and write input / output rate is higher than the preset rate benchmark, the system can identify the storage resources with excellent current performance or high idle degree, and actively direct more write requests to these resources. This resource allocation method based on dynamic perception of I / O capabilities breaks the limitations of traditional static configuration, realizes the real-time matching of write traffic and processing capabilities, reduces the congestion risk of high-load Pools, and improves the overall throughput performance and write concurrency ability. At the same time, this method requires no manual intervention, has the ability of automatic adjustment, provides a more fine-grained optimization path for the multi-Pool scheduling strategy, and helps to improve the scalability and stability of the system.

[0074] S40. When it is monitored that the data access frequency of a Pool exceeds a preset frequency threshold, increase the current write weight of the Pool whose data access frequency exceeds the preset frequency threshold.

[0075] In this embodiment, the data access frequency refers to the number of read or query requests initiated for the stored data in a certain Pool within a unit time, and it is an important dynamic indicator for measuring whether the Pool is in an access hot spot state. This indicator can be obtained through statistical analysis of data request logs, user behavior records, call frequencies of application services, etc., and it is a key parameter reflecting business access pressure and hot spot distribution. In the system scheduling logic, if the access frequency of a certain Pool remains at a high level for a long time, it can be inferred that it is carrying a high-intensity business load, such as frequently called order data, diagnostic images, hot spot logs, etc. At this time, if a static or equal-weight write strategy is still adopted, it may further exacerbate the load of this Pool, causing service blockage or response delay.

[0076] Therefore, by monitoring the data access frequencies of each Pool and comparing them with a preset frequency threshold set in the system, Pools in the access hot spot state can be dynamically identified. When the access frequency of a certain Pool exceeds this threshold, it indicates that the access pressure it undertakes in the current business environment is relatively large. At this time, the system should dynamically adjust the write weight of this Pool based on this state feature. Increasing the write weight in this scenario is not to increase the load, but to make full use of the locality of hot spot data: writing new associated data into the hot spot Pool first can improve the subsequent access hit rate and reduce the transmission delay caused by cross-Pool access, thereby optimizing the overall read-write coupling performance of the system.

[0077] In an actual system, the acquisition of the access frequency can be aggregated based on a classification statistics method according to data types (such as logs, transaction records, structured tables, etc.) to improve the accuracy of monitoring. The preset frequency threshold can be set according to historical access patterns, custom policies or automatic learning mechanisms, or can be dynamically adjusted to adapt to business fluctuations.

[0078] After it is found that the access frequency of a certain Pool exceeds the frequency threshold, its write weight can be increased according to the set ratio or segmented rule, so that this Pool has a higher write allocation probability in the data write scheduling link.

[0079] For example, the system regularly counts the total number of access requests for each Pool within a fixed time window, and groups and aggregates them according to data type, source user, or specific business module. For example, it can be counted once every 10 seconds, and each time the data access records in the past 60 seconds are counted, and the access times of each type of data on each Pool are recorded separately. The access records can be sourced from API call logs, distributed database access traces, traffic monitoring of data gateways, etc. After the statistics are completed, the total access times of each Pool are compared with the preset frequency threshold maintained by the system. If the access frequency of a certain Pool is greater than the threshold, for example, exceeding 500 times / minute, the write weight improvement mechanism is triggered. The improvement method can adopt a linear gain mode, for example, the weight is increased by 10% for every 10% exceeding the threshold; an exponential weighting method can also be used to achieve stronger response sensitivity. For example, assuming that the original write weight of a certain Pool is 0.2 and the current access frequency is 1.5 times the threshold, then its new write weight can be adjusted to 0.2×(1 + 0.5×α), where α is the adjustment coefficient, such as 0.6. The final result will be used to normalize and generate a new discrete probability distribution table, and guide the subsequent data write scheduling.

[0080] By dynamically increasing the write weight of the corresponding Pool when the data access frequency exceeds the preset frequency threshold, the system can implement a convergence strategy for read / write hotspots, actively write the newly generated data into the Pool that is already in the hot state, thereby reducing the latency caused by cross-Pool queries during future accesses. This strategy not only improves the access hit efficiency but also reduces the overall data scheduling and query costs, and is particularly suitable for application scenarios with significant hotspot focus.

[0081] S50, when it is detected that the average response time of a Pool exceeds the preset time threshold, the current write weight of the Pool whose average response time exceeds the preset time threshold is reduced;

[0082] In this embodiment, the average response time is the core performance metric for measuring the overall service ability and operating load of the Pool, referring to the average value of the time consumed from the request initiation to the return of the response result, usually including multiple latency factors such as I / O scheduling latency, disk access latency, memory cache hit rate, and network transmission time. The higher the response time, the weaker the current load processing ability or the more serious the resource bottleneck of the Pool. Using the average response time as the adjustment basis for the dynamic write policy can effectively reflect the performance changes in the actual operating environment, thereby optimizing the write allocation.

[0083] The preset time threshold is a performance benchmark set by the system based on historical baseline response times, hardware capability standards, or service quality requirements, reflecting the upper limit of the system's tolerance for acceptable response latency. If the current average response time of a certain Pool exceeds this time threshold, it means that it has entered a state of performance degradation, and continuing to perform a large number of write operations may exacerbate the blockage, resulting in a decline in global performance.

[0084] After detecting that the average response time of a certain Pool exceeds the preset time threshold, reducing the current write weight of this Pool can achieve load migration, guiding new data writes to other Pools with faster responses, thereby alleviating the processing pressure on the abnormal Pool. This mechanism has dynamic adjustability and does not rely on fixed rules, but is scheduled in real time based on performance, effectively coping with unpredictable system load fluctuations.

[0085] This strategy works together with other dynamic metrics (such as available space, access frequency, etc.) to form a multi-dimensional write adjustment model, making the use of storage resources more intelligent and balanced.

[0086] For example, the system periodically collects the data request response time records of each Pool within the statistical window period, including successfully completed data write and read tasks. For each response record, extract its time-consuming field and accumulate the sum, and then calculate the ratio of the total response time to the number of successful responses to generate the average response time data for the current period. The statistical period can be set to 1 minute, and 1000 response records are collected within each period. The system generates the average response time by dividing the total time by the number of responses, such as 95 milliseconds. If this value exceeds the preset time threshold set by the system (for example, 80 milliseconds), the weight adjustment mechanism is triggered. The reduction of the write weight can adopt a linear attenuation strategy, for example, reducing the current weight by 10% for every 10% exceeding the threshold; it can also be discretely adjusted based on multiple levels of segmentation to avoid system instability caused by too rapid fluctuations in the write weight. After adjustment, the system normalizes and updates the current write weights of all Pools to generate a new probability distribution table for subsequent write scheduling logic. In actual deployment, the response time measurement can be obtained through the read and write service logs at the application layer, the storage device monitoring interface, or the performance statistics module at the system I / O layer to ensure comprehensive and accurate data.

[0087] By real-time monitoring of the average response time of each Pool and comparing it with the preset time threshold of the system, when a certain Pool shows a downward trend in response performance, its write weight is dynamically reduced, which can effectively avoid the further deterioration of the performance bottleneck. This mechanism realizes the automatic avoidance of write traffic, enables the redistribution of data among Pools with higher performance, and improves the overall system throughput and response efficiency. At the same time, in scenarios where there are significant performance differences among multiple Pools, load migration can be performed according to real-time performance, avoiding global blocking or data retention caused by write overload, and significantly enhancing the system's adaptive scheduling ability.

[0088] S60, if there is a Pool in an abnormal state, then set the current write weight of the Pool in the abnormal state to zero;

[0089] In this embodiment, the abnormal state usually refers to a situation where a certain storage Pool fails to process or respond to write requests normally during operation due to problems such as hardware failures, network disconnections, or performance degradation. Timely identification and handling of Pools in an abnormal state are key means to improve the stability of the storage system and the write fault tolerance ability. In a distributed storage architecture, a Pool is usually bound to one or more physical storage devices, including storage arrays, disk groups, or solid-state drive units. Once these underlying resources are abnormal, if write requests continue to be sent to them, it will not only cause data write failures but also may lead to data corruption or system-level timeouts, further affecting the upper-layer services.

[0090] The write weight is a reference value for measuring the currently acceptable write traffic of a certain Pool, and it directly determines the probability of the Pool in subsequent write target selection. When it is found that a Pool is in an abnormal state, immediately setting its write weight to zero is equivalent to removing the Pool from the scheduling policy, ensuring that data will not continue to be written to risky storage nodes. This operation has high real-time performance and protection, and is an important part of realizing write disaster tolerance.

[0091] The determination of the abnormal state can be comprehensively identified by combining multiple sub-indicators. For example, by obtaining disk SMART warning events through underlying hardware monitoring, capturing heartbeat loss signals through network monitoring, and detecting a sharp increase in response time through performance detection, etc., and further classifying them into types such as hardware abnormalities, network abnormalities, or performance abnormalities. After classifying and marking the abnormal state, it also provides a judgment basis for subsequent recovery scheduling or state rollback.

[0092] For example, during operation, the system continuously monitors the running status of all Pool-bound resources. Once it detects that the underlying storage device of a certain Pool has serious SMART attribute alarms, such as the continuously increasing reallocated sector count or the abnormally elevated interface error rate, the system will immediately mark that Pool as in a hardware exception state; if a certain Pool loses heartbeats several times in a row, it indicates that the Pool may have been disconnected or out of the control of the management node, and it will be marked as a network exception; and when the average response time of a certain Pool exceeds a certain multiple of the preset time threshold, such as more than twice, and lasts for multiple statistical periods, it will be marked as a performance exception. After any of the above abnormal states is determined, the system will trigger the action of resetting the write weight to zero. The process of resetting the write weight to zero does not directly operate on the storage hardware, but is set through the weight control table in the system, updating the current write weight field of the abnormal Pool to zero and recording the current abnormal type in the abnormal identification field. When the write scheduling engine selects the target Pool in the next cycle, it will skip all Pools with a weight of zero to ensure that abnormal resources are no longer selected. In addition, the system can also set the weight recovery determination logic. After the abnormal Pool completes the status repair, such as re-establishing the heartbeat or confirming the hardware stability through background consistency checks, the write weight will be gradually restored to avoid the Pool encountering high-intensity writes at the initial stage of recovery.

[0093] By introducing the abnormal state recognition and write weight reset mechanism, it is possible to achieve timely isolation of faulty storage nodes, prevent high-risk Pools from continuing to participate in write scheduling, and thus ensure the reliability and stability of the data write path in the system. It significantly improves the fault tolerance of the distributed storage system. Especially in a large-scale environment with a large number of nodes and intensive business loads, it can avoid abnormal nodes from dragging down the overall performance or causing interruption risks. In addition, through abnormal classification and marking, it also provides support for the subsequent state recovery of the system, making the entire scheduling system more resilient and self-healing.

[0094] S70, convert the current write weight of each Pool into a probability value, and construct a discrete probability distribution table based on the probability value;

[0095] In this embodiment, the current write weight represents the adaptability of each storage Pool to write tasks at a given moment. To implement a more flexible and real-time responsive data allocation strategy, the current write weight can be converted into a normalized probability form to construct a target selection mechanism for dynamic adjustment of the system state without introducing complex sorting or manual allocation.

[0096] The current write weight can be generated by driving multiple metrics, such as dimensions like available storage space, I / O rate, response time, data access frequency, etc. In different time periods, different Pools may simultaneously meet multiple positive or negative adjustment conditions. To quantify the comprehensive scheduling priority of each Pool, it is necessary to merge the adjustment impacts of each condition on the current write weight. Among them, the processing method of algebraic sum has strong interpretability and flexibility. For a certain Pool, if it meets both the conditions for increasing the weight (such as high I / O rate) and the conditions for decreasing the weight (such as insufficient space), the respective adjustment amplitudes can be represented in positive and negative numbers, and then calculate their algebraic sum as the final weight adjustment value.

[0097] The core purpose of constructing the discrete probability distribution table is to convert the above-adjusted weights into probability intervals that can be used for pseudo-random scheduling. For this purpose, it is necessary to normalize the current write weights of all Pools, that is, divide the weight value of each Pool by the sum of the weight values of all Pools to obtain its normalized probability value. These probability values can be mapped into numerical intervals, forming continuous and non-overlapping segments to constitute a complete discrete probability distribution table for the pseudo-random algorithm to call.

[0098] For example, the system first maintains a dynamic variable of the current write weight for each Pool, and this variable is adjusted in real time during the scheduling cycle according to the monitoring results of multiple metrics. When a certain Pool meets multiple adjustment conditions, the system will calculate its corresponding adjustment amplitude according to the increment or decrement value set by each condition respectively. For example: if the I / O rate is higher than the threshold, it will increase by 5, and if the response time is too long, it will decrease by 3. When both conditions are met, the total adjustment value is +2. This synthesis result will be added to the weight value retained in the previous cycle and limited between the preset minimum and maximum ranges to prevent extreme bias. Next, the system traverses the current write weights of all Pools and sums them to obtain a total weight value. Then, divide the current write weight of each Pool by the total weight value to obtain the normalized probability value of this Pool. For example, if the total weight is 100 and the weight of a Pool is 25, then its probability is 0.25. The system constructs a continuous interval between 0 and 1 based on these probability values, and each Pool occupies a section corresponding to its probability value. Finally, a discrete probability distribution table is formed, with each Pool corresponding to an interval range. This probability distribution table will be called in the subsequent target Pool selection step. When a write request arrives, the system generates a random number between 0 and 1 through a pseudo-random number generator and determines the target Pool according to the interval where this random number falls.

[0099] By introducing a mechanism that converts the current write weight into a probability value and constructs a discrete probability distribution table, a refined Pool selection strategy based on dynamic multi-factor adjustment is realized, effectively replacing the traditional fixed weight allocation method. This mechanism takes into account the comprehensive impact of multiple performance dimensions and supports high-dimensional resource status evaluation, not only improving the fairness and real-time performance of target selection, but also avoiding the imbalance problem caused by a single metric drive. At the same time, by using the algebraic sum method to fuse the adjustment impacts of multiple conditions, the system can more accurately reflect the current comprehensive ability status of each Pool.

[0100] S80, determine the target Pool through a pseudo-random number generator according to the discrete probability distribution table;

[0101] In this embodiment, to achieve efficient and dynamically adaptable data allocation, a probability-based Pool selection strategy is adopted, which can significantly reduce the computational overhead and rule conflicts in the resource scheduling process. Among them, the core logic of selecting the target Pool through a pseudo-random number generator is to map the write weight of each Pool in the system to a specific probability interval by using the previously constructed discrete probability distribution table, and further transform the resource scheduling into a triggering process of probability events.

[0102] The discrete probability distribution table is composed of multiple non-overlapping probability intervals, each interval corresponding to a Pool, and the current write weight of this Pool determines the interval length on the probability axis after normalization conversion. The preset numerical range is usually a closed interval, such as from 0 to 1, and this interval is divided into multiple segments, each segment corresponding to a Pool, and its length is proportional to the normalized probability value of the Pool.

[0103] The pseudo-random number generator is a deterministic finite state machine, which can output uniformly distributed pseudo-random real numbers after being initialized with a seed value, and its output can be used to simulate probability events. When mapping the generated random number to the discrete probability distribution table, the system sequentially traverses the probability intervals corresponding to each Pool to determine whether the random number falls into the currently traversed interval. If it falls into the probability interval of a certain Pool, then this Pool is selected as the target Pool for this data write, thus completing the determination process of the target Pool.

[0104] This mechanism essentially realizes the probability control of the write target and dynamically adaptable scheduling with low computational complexity, without relying on complex real-time sorting and scoring logics, and is an effective alternative to the traditional absolute priority-based mechanism.

[0105] For example, the system maintains a discrete probability distribution table, which is generated based on the normalization of the current write weights of all Pools. The specific method is as follows: sum up the current write weights of all Pools, and then divide the write weight of each Pool by the sum to generate a normalized probability value. The system constructs the start and end points of the intervals in a unified order (such as ascending Pool numbers) so that each Pool has a continuous probability interval in the range from 0 to 1.

[0106] When there is a write request that needs to allocate a target Pool, the system calls a pseudo-random number generator to generate a floating-point number between 0 and 1, and this random number serves as the mapping input value. The system traverses the probability distribution table sequentially starting from the first Pool to find the interval where this random number is located. Once the interval of a certain Pool is matched, that Pool is immediately selected as the target Pool for the current write request, and the traversal ends.

[0107] To enhance the robustness of the system, a fault tolerance mechanism can also be set. If in extreme cases, no interval is matched due to data anomalies (such as the interval sum being less than 1 because the probability distribution is not updated in time), the default policy is adopted to select the Pool with the highest current write weight as the target Pool to ensure that the write process does not interrupt.

[0108] By constructing a discrete probability distribution table and combining it with a pseudo-random number generator for target Pool selection, dynamic write allocation can be completed without the need for real-time full-scale sorting. This not only effectively reduces the complexity of the system scheduling logic but also maintains the resource scheduling efficiency in high-throughput scenarios. In addition, the mechanism based on probability mapping rather than hard priorities helps to balance the load pressure among different Pools and shows better balance in the resource usage distribution.

[0109] S90, write the data to be stored into the target Pool.

[0110] In this embodiment, the process of writing the data to be stored into the target Pool is the final execution link of the entire storage resource adaptation mechanism. Its essence is to establish an effective data channel between the target storage resource node and the data source and complete the write operation. The target Pool usually refers to the data receiving node finally determined by the system according to a series of index monitoring and weight adjustment logics during the current scheduling period, which may be a single storage node or a storage cluster containing multiple physical disks or logical volumes. This process is not simply issuing a write instruction but involves operations in multiple dimensions such as adaptation control, data format standardization, cache processing, exception response, and status feedback.

[0111] In specific implementation, the system first generates a write request that conforms to the target storage specification according to the access protocol and device type of the target Pool, including but not limited to operations such as constructing a compliant data block format, setting the corresponding partition path, allocating a write buffer, and scheduling write threads. For a Pool with parallel write capabilities, the system also needs to perform data slicing and concurrent channel scheduling control. If the target Pool is configured with a pre-write verification mechanism (such as a data integrity identifier or CRC check), it is also necessary to generate and write the corresponding markers before data writing.

[0112] In addition, to enhance the fault tolerance of the system, the current write queue length, remaining cache space, and other statuses of the target Pool can be quickly detected before data writing. If it exceeds the preset threshold, switch to the standby Pool or trigger a retry mechanism. After data writing is completed, the system needs to obtain the status code or write confirmation flag returned by the target Pool and update the execution record of the internal scheduling module to provide a feedback basis for subsequent dynamic weight adjustment.

[0113] A specific implementation method is to encapsulate the data writing operation as an asynchronous call and push the write task to the write interface corresponding to the target Pool through the task scheduling thread pool. Before the write task starts, the system loads the write configuration of the Pool through the storage adaptation module, including the path structure, I / O queue depth, write protocol type (such as S3, NFS, iSCSI), etc., and pre-allocates cache space for data staging before task execution.

[0114] For large-scale write tasks, the system can first segment the data to be written in the local cache and determine the segmentation granularity according to the bandwidth threshold configuration of the target Pool. During the execution of the write task, use a non-blocking I / O interface to submit write requests and monitor the return status in real time. If the write fails or times out, the system triggers the corresponding fallback strategy according to the type of failure (such as target unreachable, disk full, disk failure, etc.), such as retry, degradation, or switching to the standby Pool. All records of successful or failed writes will be synchronously written to the log system to support subsequent performance analysis and exception tracing.

[0115] In a deployment scenario with a multi-level cache or write-ahead log mechanism, the data is first written to the intermediate cache layer or the write-ahead log area, and then batch-written to the physical device when the system is idle, thereby optimizing the I / O pressure and latency fluctuations under high concurrency.

[0116] Example illustration: In the field of fintech business, a real-time risk control system performs high-speed writing on a large amount of heterogeneous data such as transaction logs, user behaviors, and risk rule matching results. This system is deployed on a distributed multi-Pool storage architecture, where each Pool is bound to storage devices with different performance levels. For example, some Pools are deployed on NVMe SSD nodes, and some are deployed on hybrid HDD arrays. During the peak period of daytime trading, the risk assessment model is frequently triggered for update, resulting in a significant increase in the writing pressure of some Pools. At this time, by monitoring the available storage space, read-write input / output rate, average response time, and data access frequency of each Pool, the system automatically identifies the performance bottleneck of the hot Pool and dynamically reduces its writing weight. At the same time, the system increases the writing weight of the low-load Pool to prevent the overloaded node from continuously writing and affecting the overall transaction throughput rate. Finally, based on the probability distribution and pseudo-random number selection mechanism constructed by the dynamic weight, the selection process of the target Pool is ensured to be elastic and load-aware, achieving efficient adaptation of the writing resources without introducing high computational overhead.

[0117] In a medical and health data management system, there is continuous writing and archiving of a large amount of medical record records, diagnostic images, and real-time vital sign data. This system also adopts a multi-Pool distributed storage structure, where different Pools correspond to the archiving storage requirements of different hospital departments or different time periods. The system dynamically identifies changes in the access pattern and increases the writing weight of the Pool where such data is located by regularly monitoring the writing pressure and access popularity of each Pool. For example, the access frequency of data related to the ICU has increased significantly during a specific sudden epidemic. If the response time of some storage nodes increases due to long service life, the system automatically senses that its average response time exceeds the set threshold and reduces the writing weight of this Pool to ensure low latency and high stability during the writing process of critical patient data. The selection of the target Pool is based on a discrete probability model generated by weight normalization and a pseudo-random selection strategy, effectively avoiding repeated hits on local high-load nodes and achieving a balance between data writing stability and equipment service life in the medical system.

[0118] In the smart city transportation system, a large amount of images and structured data generated by traffic cameras and geomagnetic sensors need to be written into the backend system in real time for road condition analysis. The data traffic in different urban areas varies significantly. Cameras in busy sections have a high frequency and large amount of data, while the data volume in remote areas is relatively low. The system deploys multiple Pool storage resource pools for regional management, and continuously monitors the space occupancy and input / output rate changes of each Pool. In the high-density area, the access frequency and write rate of the Pool will increase simultaneously during the rush hours. The system adjusts the write weights of these Pools in real time and reflects it in the target Pool selection, so that write requests can be assigned to nodes with remaining processing capacity with a higher probability, thus effectively balancing the storage loads in busy and idle areas and ensuring the data write quality at key traffic nodes.

[0119] By adapting the write strategy according to the current state of the target Pool and combining with the asynchronous non-blocking scheduling mechanism, the single write latency can be significantly reduced, and the write stability of the system in high-concurrency scenarios can be improved. At the same time, the introduced status confirmation mechanism ensures that data write operations can be monitored and traced, supporting the entire dynamic weight feedback closed loop, so as to achieve more accurate Pool performance perception and subsequent scheduling strategy adjustment.

[0120] The present invention relates to the technical field of data storage, and can be applied to business scenarios such as medical health and fintech. It discloses a method for adapting write operations to multiple Pool storage resources, including: monitoring the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool; dynamically adjusting the current write weight of the Pool according to the monitoring results; converting the current write weights of all Pools into probability values to construct a discrete probability distribution table; selecting the target Pool according to the probability distribution table through a pseudo-random number generator; and writing the data to be stored into the target Pool. By continuously monitoring the running state of each Pool and adjusting the write weight in real time, the present invention avoids the problem of rigid resource allocation under the fixed weight strategy; combines the discrete probability distribution table constructed based on the normalized weight and the pseudo-random selection mechanism, making the selection of the target Pool have a certain load balancing flexibility, and can more effectively guide the data write load to the Pool with lighter load and better performance; at the same time, through the abnormal state detection mechanism, the unavailable or abnormally performing Pools are actively isolated, further enhancing the robustness and write fault tolerance of the system, thus significantly improving the write efficiency, system stability and resource utilization rate in the high-frequency trading environment in the fintech business field and the high-concurrency data recording scenario in the medical health business field.

[0121] In one embodiment, the above step S10 includes:

[0122] S101. Perform millisecond-level periodic scans on the storage devices of each Pool to generate the available storage space value;

[0123] S102. Based on the sliding window statistical strategy, count the total number of read and write operations of each Pool within a preset time window to generate the read and write input / output rate value;

[0124] S103. Measure the end-to-end response time of the data requests of each Pool and deduct the preset network transmission baseline delay value to generate the corrected average response time value;

[0125] S104. Classify and count the data access times of each Pool per unit time according to user information, transaction records, and log data types to generate the data access frequency value.

[0126] In this embodiment, the acquisition of available storage space depends on the accurate statistics of the capacity occupancy of physical or virtual storage devices in each Pool. By calling the low-level storage driver interface or obtaining the measurement data of the remaining space and total capacity through the storage management controller within each periodic scan cycle, the ratio of the two is calculated as the current available storage space percentage. The time accuracy of the periodic scan can be refined to the millisecond level, mainly to meet the response timeliness requirements for resource exhaustion warnings in high-concurrency data stream environments.

[0127] The monitoring of the read and write input / output rate is based on the sliding window statistical mechanism. The number of operation records within the window boundary is divided by the window duration to obtain the IOPS (Input / Output Per Second) value per unit time. The sliding window can advance continuously based on timestamps to ensure that trend-smooth data can still be obtained under rapidly changing loads. The record sources can include the IO statistical module at the operating system level or the IO scheduling system at the middleware layer.

[0128] The measurement of the average response time adopts the end-to-end observation method, that is, the full-process response delay is from the time point when the client initiates a request to the response time point when the request is completed. To eliminate the interference of network-level fluctuations, the preset network transmission baseline delay value is deducted in the calculation. This baseline value is usually obtained through the deployment of initial calibration tests and can be configured according to different deployment topologies of the business data center. The finally obtained corrected response time can better reflect the true processing performance of the internal devices of the Pool and is applicable to the performance comparison of heterogeneous storage devices.

[0129] The acquisition of data access frequency is based on the typed classification of access records. Common classification criteria include the user identification of the request source, the types of data accessed (such as logs, transaction records, pictures, videos, etc.), and the business tags to which the data belongs. Within a unit time period, these classification requests are counted. The obtained data access frequency not only reflects the overall activity of the Pool but also supports fine-grained hotspot identification and business trend analysis. Integrating a log proxy or a data tracing and collection mechanism at the application layer can effectively improve the accuracy of classification and statistics.

[0130] The above four indicators together constitute a multi-dimensional monitoring model for the operating status of each Pool. In actual deployment, the monitoring tasks of these four types of indicators can be executed in parallel, independent of each other but collaborating to form a resource load assessment system.

[0131] In the actual deployment environment, based on a unified resource monitoring service module, the storage device status of all Pools can be connected to the data collection bus of this module. For the acquisition of available storage space, the "df - h" under Linux can be called or the space information in the / sys / block directory can be directly read, and the remaining capacity and total capacity can be collected regularly through a custom script. The implementation of read / write input / output rate can integrate the built-in performance counting modules in distributed systems such as ceph, glusterfs, and hdfs, or deploy an Exporter on an open-source monitoring platform such as Prometheus for periodic collection and sliding window operation. For the correction of the average response time, a lightweight TraceAgent can be inserted into the client request link to record the timestamps of the complete request response path, synchronize the network latency benchmark value through the configuration center, and summarize the data after subtraction processing at the collection end. The classification and statistics of data access frequency can tag the traffic with the help of a data access gateway, and classify and summarize logs or business events in real time through middleware such as Kafka and Fluentd, map them to different Pools according to the data source and data structure, and generate frequency statistics data. The system can configure a unified indicator collection period, such as 100ms as a common refresh window, to ensure the timeliness consistency of various data in reasoning and decision-making.

[0132] In this embodiment, by establishing a multi-dimensional monitoring model including space, rate, response, and access frequency, a full-range dynamic control of the usage status of storage resources is achieved, and a more real-time and accurate weight adjustment basis is constructed before data writing and allocation. It can respond to the fluctuations of the storage environment within milliseconds, ensure that the write operation preferentially tilts towards the target areas with high efficiency, stability, and reasonable load, thereby improving the overall write efficiency of the system, reducing the probability of hotspot conflicts, delaying the aging of storage devices, and ultimately ensuring the performance stability and business continuity of the data writing process.

[0133] In one embodiment, after step S10 and before step S30, the following steps are further included:

[0134] S105, obtaining the total storage capacity of all Pools and statistically calculating the average historical storage growth rate of each Pool;

[0135] S106, based on the total storage capacity and the average historical storage growth rate, determining a predicted value of the global storage occupancy rate for a preset future period through a linear extrapolation module;

[0136] S107, setting a preset space threshold as the difference percentage between the total storage capacity and the predicted value of the global storage occupancy rate;

[0137] S108, obtaining the peak data of the historical read / write input / output rate of all Pools and determining the global average peak rate based on the peak data of the historical read / write input / output rate;

[0138] S109, monitoring the comprehensive load rate of the current system and adjusting the global average peak rate according to the comprehensive load rate to generate the preset rate benchmark.

[0139] In this embodiment, after monitoring the key operating indicators of each Pool, in order to improve the system's ability to predict the future resource status and the forward-looking nature of the write decision, it is necessary to introduce a dynamic perception mechanism for the global storage and system load trends.

[0140] First, obtain the total storage capacity of all Pools. This operation is used to determine the total amount of data that the entire system can physically carry. The total storage capacity is usually composed of the sum of the physical spaces of the storage nodes bound to each Pool and can be directly aggregated through the system-level metadata collection interface. At the same time, statistically calculate the average storage growth rate of each Pool over a past period of time, that is, the historical average of the data write speed per unit time. This indicator can be obtained by recording the storage increment per unit time and calculating the multi-period average value, which reflects the trend of data growth in the current system. Using these historical increment data as input, a linear extrapolation module is used to predict the global storage occupancy rate within a preset future period based on the existing growth trajectory. The linear extrapolation module can achieve rolling prediction based on a sliding time window, taking into account both short-term fluctuations and long-term trends.

[0141] After generating the predicted value of the global storage occupancy rate, it is necessary to perform a difference operation between it and the total storage capacity to obtain the capacity space that has not been expected to be occupied at the system level. Converting this difference to a percentage of the total capacity can be used as the dynamically adjusted space threshold to replace the traditional fixed threshold setting strategy. The lower this percentage, the less redundant space there is in the system, and the write weight adjustment and load migration should be carried out in advance.

[0142] On the other hand, in order to dynamically generate a rate benchmark for adjusting the write weight, it is necessary to obtain the peak data of the historical read / write input / output rates of all Pools. These peak data reflect the upper limit of the processing capacity of each Pool under extreme load conditions. By statistically calculating the mean of each peak data and generating a global average peak rate, it can be used as a reference value for the maximum load capacity of the overall system under the past performance. To adapt to the changes in the current system state, it is necessary to monitor the comprehensive load rate of the system in real time, and this load rate can be calculated by combining multi-dimensional metrics such as CPU usage, I / O utilization, and memory usage. According to the real-time load rate, the above average peak rate is weighted and adjusted to generate a preset rate benchmark to guide the subsequent write decision-making process. This process ensures that the rate benchmark changes dynamically with the system pressure during runtime and has the ability to adapt.

[0143] There are various ways to achieve the extraction of the total storage capacity and growth rate. One way is that the storage nodes bound to each Pool periodically report the used space information, calculate the increment within a unit time, and use the window sliding technology to extract the stable trend. The linear extrapolation module can be based on simple statistical models such as regression models and moving average models, or it can integrate lightweight time series prediction algorithms such as ARIMA or exponential smoothing models to improve the prediction accuracy without increasing the system burden.

[0144] For the generation of the global peak rate, the system can maintain the maximum throughput record of each Pool within every 5 seconds in the recent period of time, retain the peak data within N cycles and then take the average. This process does not affect the real-time processing path and is completed by an asynchronous calculation thread. The acquisition of the comprehensive load rate can integrate the performance monitoring modules of existing operating systems, such as the / proc file system of Linux or tools like iostat, and calculate a weighted metric after regular sampling to correct the global average peak rate.

[0145] In addition, the dynamically adjusted space threshold and rate benchmark will be injected into the write policy scheduling module in real time as the trigger basis for the dynamic adjustment of the write weight of each Pool. The entire process has real-time performance and traceability, and the prediction period, window size, and adjustment frequency can be controlled parametrically.

[0146] This embodiment constructs dynamic thresholds and benchmark metrics through historical growth rates and current loads, which can more accurately reflect the actual operating state and future trends of the storage system, thereby implementing a more forward-looking write traffic control mechanism. It not only effectively avoids resource skew or performance bottlenecks caused by over-reliance on fixed parameters, but also improves the flexibility of the write policy and the overall response ability of the system. Especially in scenarios where the data scale expands rapidly or the business load fluctuates violently, it can significantly reduce the risks of system write conflicts, load concentration, and resource waste.

[0147] In one embodiment, after the above step S10 and before step S30, the following steps are further included:

[0148] S110, access the event calendar data of the external service module, and parse the event type code and time interval in the event calendar data;

[0149] S111, filter out the events with the event type code being the preset high-concurrency event type, and use the time interval of the corresponding events as the system-level peak periods;

[0150] S112, compare the current time with the start and end times of the system-level peak periods;

[0151] S113, if the current time is within the time interval of any system-level peak period, increase the preset frequency threshold to the preset emergency threshold;

[0152] S114, if the current time is outside the time intervals of all system-level peak periods, restore the preset frequency threshold to the preset normal threshold.

[0153] In this embodiment, after monitoring the core performance parameters of each Pool (including available storage space, read / write input / output rate, average response time, and data access frequency), in order to further enhance the response ability of the write path selection mechanism to changes in the external service environment, a dynamic parameter adjustment mechanism based on the event calendar data provided by the external service module is introduced.

[0154] The event calendar data usually comes from a business scheduling system or an application event control platform co-deployed with the storage system, and such modules can be found in large transaction systems, task scheduling systems, or content publishing platforms. For example, in a financial service scenario, the event calendar may be provided by a clearing system or a business warning system to record high-intensity business periods such as end-of-month reconciliations and high-frequency trading days; in a medical and health service scenario, it may be output by a reservation center or an imaging archiving module in a hospital HIS system to record large-scale health screening days or specific data audit cycles.

[0155] The event type code in the event calendar refers to the identification of specific types of high-concurrency business behaviors. For example, "TRD_HOT" can be used to identify high-concurrency trading behaviors, and "DATA_SYNC" can identify batch data synchronization behaviors, etc. By parsing this code and combining the time interval information, the system can accurately construct a peak period model associated with the current storage load. To achieve this, the system needs to establish an event type mapping table and time interval parsing rules, and support updates and synchronizations through configuration files or API methods.

[0156] During the screening process, the system will identify events in the event calendar that may cause fluctuations in storage I / O pressure according to a preset list of high-concurrency event types, and mark the corresponding time intervals as system-level peak periods. A system-level peak period can be formed by the union of time intervals of multiple events, and the system needs to have the ability to merge overlapping intervals and detect conflicts.

[0157] The subsequent time comparison operation is completed through the timestamp synchronization mechanism. The system obtains the current time in real time and judges the inclusion relationship between it and all time intervals of the system-level peak period. To avoid judgment errors, it is recommended to introduce a buffer window (such as judging to enter the peak period 10 minutes in advance) in this step to ensure that the adjustment of the frequency threshold is completed before the peak event actually arrives, with forward-looking and stability.

[0158] Once it is determined that the current time is within the system-level peak period, the frequency threshold is adjusted from the normal value of the basic configuration to the emergency threshold. The emergency threshold can be understood as the relaxation boundary of the system's load response ability, and its value is usually determined by analyzing historical I / O response characteristics. For example, during the "Double 11" promotion period on an e-commerce platform, the frequency of log write requests can be up to 3 times that of the normal period, and the system can configure the emergency threshold to be between 250% and 300% of the normal value. The application of the emergency threshold ensures that hot data can be preferentially imported into the Pool with strong load-bearing capacity, improving the overall response efficiency and avoiding I / O blocking during peak periods.

[0159] If the current time is outside all peak periods, the system automatically reverts to the normal frequency threshold to avoid unnecessary tilting of the storage load distribution caused by continuous high-threshold configuration, thus ensuring balanced resource utilization and stable read / write paths.

[0160] In one embodiment, the system periodically pulls event calendar data from an external business module, which can be a business process management system, a task scheduling platform, or an enterprise service bus. The event calendar data is stored in the form of JSON, XML, or a database. The system parses fields such as event_type_code and time_range to form a standardized event time mapping structure. Through a configuration file or a policy table, the system pre-defines which event_type_code belongs to the high-concurrency type. The filtering rule is matched in the form of a whitelist, such as ET101 (payment peak), ET202 (hospital registration), ET999 (government service open day), etc. When the match is successful, the corresponding time_range is added to the peak period list. The system uses a scheduled task (such as once a minute) to determine whether the current time is within any of the above peak period intervals. If a hit occurs, the frequency threshold adjustment logic is executed. The threshold adjustment is achieved by accessing the system parameter control module, which switches the original threshold identifier to the corresponding value in the emergency mode; after leaving the peak period, it is restored to the normal value. This adjustment does not change the historical policy, but injects the current policy control module in a hot update manner, which is applicable to the scenario of uninterrupted business operation. To prevent jitter caused by frequent switching, a buffer duration can also be set. For example, after entering the peak period, it is maintained for at least X minutes before making a determination to avoid boundary jitter.

[0161] In this embodiment, by constructing a threshold dynamic adjustment mechanism for event perception, the real-time response ability of the data writing policy to high-concurrency events in external services is achieved. The system no longer relies on static rules to judge the high or low access frequency of the Pool, but can dynamically set the frequency threshold based on foreseeable future business events, thereby effectively avoiding Pool write skew or performance degradation caused by improper setting of fixed thresholds. The threshold increase strategy improves the system's tolerance to business peaks, and the threshold recovery strategy ensures the balanced allocation of resources in the non-peak period of the system, thus taking into account both performance stability and resource utilization efficiency.

[0162] In one embodiment, after the above step S10 and before step S50, the following steps are further included:

[0163] S115, collecting health status indicators of all storage devices bound to the Pool, including disk SMART status parameters, solid-state drive wear percentage, and disk array degradation event count;

[0164] S116, performing a standardized score on the health status indicators of each storage device to generate a health score for each storage device;

[0165] S117, determining the mean value of the health scores of all storage devices and using the mean value as the global hardware health score;

[0166] S118. When the initialization of the storage device is completed or within a preset calibration period, send a standardized read / write test instruction to all storage devices bound to the Pool;

[0167] S119. Measure the time interval from the issuance of the standardized read / write test instruction to the return of an operation completion confirmation signal by the storage device as the reference response time of a single storage device;

[0168] S120. Aggregate the reference response times of all storage devices and use the average value of the reference response times of all storage devices as the global reference response time average;

[0169] S121. Determine a redundancy time compensation value based on the global hardware health score and the global reference response time average;

[0170] S122. Set a preset time threshold as the sum of the reference response time average and the redundancy time compensation value.

[0171] In this embodiment, based on monitoring the operating state of the storage system, by introducing a joint modeling mechanism for the health state indicators and response performance baselines of storage devices, the dynamic optimization and adjustment of the time threshold are further realized. The core of this mechanism lies in comprehensively perceiving the operating state and service capabilities of the underlying storage hardware, and combining the standardized test feedback to construct a flexible performance threshold judgment system. First, collecting the hardware health state indicators of all storage devices bound to the Pool is realized based on the transparent design of the storage hierarchy, including obtaining SMART (Self-Monitoring Analysis and Reporting Technology) parameters to reflect the key warning signals of mechanical hard disks, such as the reallocated sector count, read error rate, etc.; for solid-state drives, the wear percentage indicator is obtained to measure the consumption of the service life of the flash memory medium; at the same time, the disk array (RAID) degradation event count indicator is also introduced to judge whether there is a weakening or loss of fault tolerance ability in the redundant storage subsystem. These indicators together constitute the structured input of the operating reliability of the storage device.

[0172] Next, in order to uniformly incorporate different types of health indicators into the quantization framework, standardize the scoring of the above hardware indicators. During the scoring process, strategies such as linear normalization, segmented scoring, or multi-factor weighting can be adopted to convert multi-source health information into comparable health scores, thereby giving a stability portrait to each device. The standardized scoring mechanism supports rule configuration according to business scenario differences. For example, in the financial system, the tolerance for RAID degradation is lower, while in the cold data archiving scenario, more attention is paid to the SSD wear level.

[0173] Based on the health scores of each storage device, a global hardware health score is further aggregated, usually using arithmetic mean, weighted mean, or weighted method based on the coefficient of variation, depending on different application scenarios. The global health score, as a cross-sectional reflection of the stability and reliability of the entire system, will be used for subsequent dynamic correction calculations of performance thresholds.

[0174] Subsequently, in order to obtain a basic reference for the true response ability of the storage device, during the system initialization or within a preset calibration cycle during operation, a standardized read-write test instruction is actively sent to all storage devices bound to the Pool. This instruction is a preset sequential write or read task, avoiding business load interference. By accurately measuring the time interval between the issuance of the instruction and the device's return of the confirmation signal, the benchmark response time of each device is obtained, reflecting its minimum operation delay ability. This operation avoids measurement deviations caused by differences in business models and is an important basis for constructing the global performance baseline.

[0175] After that, the system aggregates the benchmark response times of all devices and calculates their arithmetic mean as the global benchmark response time mean, which is used to characterize the reference level of the system's median response ability. This mean can be updated with the calibration cycle and has timeliness and dynamic adaptability.

[0176] After having the two core indicators of health score and response mean, the system introduces a calculation mechanism for the redundant time compensation value. This compensation value, as an extended item of the response time threshold, reflects the system's tolerance for performance fluctuations and hardware risks. The calculation method can be a functional relationship between the global benchmark response time mean and the health score. For example, redundant time = mean × (1 - health score / 100). When the health score is low, the system response tolerance threshold is correspondingly enlarged.

[0177] Finally, the system sums the benchmark response time mean and the redundant time compensation value to construct the preset time threshold for the current operation cycle. This threshold is used to subsequently determine whether there is a performance anomaly. If the average response time of a certain Pool exceeds this threshold, it will trigger the identification of the performance anomaly state and dynamically adjust its write weight. This mechanism ensures that the system has dynamic fault tolerance and resource adaptation capabilities under different hardware states.

[0178] In this embodiment, by collecting multi-dimensional device health indicators and actual response performance, the system can dynamically construct an adaptive time threshold without relying on static configuration. When the device ages, its performance degrades, or it operates abnormally, it can be promptly reflected in the threshold update, improving the timeliness and accuracy of performance anomaly detection and avoiding continuing to write data to a Pool with performance risks. This not only improves the stability of the overall data write path but also reduces the misjudgment probability of triggering exception handling due to slow response, improving the resource utilization rate and fault controllability of the overall storage system.

[0179] In one embodiment, the above step S60 includes:

[0180] S601, when it is detected that the storage device bound to the Pool triggers a hardware fault signal, it is determined that the Pool is in a hardware abnormal state;

[0181] S602, when it is detected that the Pool continuously loses heartbeat signals for a preset number of times, it is determined that the Pool is in a network abnormal state;

[0182] S603, when it is detected that the average response time of the Pool exceeds a preset multiple of the preset time threshold, it is determined that the Pool is in a performance abnormal state;

[0183] S604, interrupt the reception of the write request of the Pool in the abnormal state, set the current write weight of the Pool in the abnormal state to zero, and mark the corresponding abnormal state type.

[0184] In this embodiment, real-time identification and weight resetting processing are performed on the abnormal state of the storage Pool, constituting the core fault tolerance mechanism in the write policy. This mechanism aims to ensure the data write stability and reliability of the overall system, and it is necessary to carefully monitor and distinguish the abnormal type, trigger condition, handling process, and subsequent records.

[0185] The determination of the hardware abnormal state depends on the self-check mechanism of the underlying storage device and the hardware monitoring signal. Among them, disk SMART (Self-Monitoring, Analysis, and Reporting Technology) is an industry-standard hardware self-monitoring protocol for continuously outputting disk health parameters; the array degradation event indicates that one or more hard disks in the RAID structure have failed but have not been repaired; and the SSD wear level is an important indicator to measure the service life of NAND flash memory. Setting the wear level upper limit as the preset wear threshold can be set based on manufacturer data or operation and maintenance experience, such as common relative values of 90%, 95%, etc. When any of the above indicators reaches the fault determination condition, it is considered that there is an irreparable fault in the hardware environment of the Pool, and it needs to be isolated immediately.

[0186] The network abnormal state is judged based on the heartbeat signal mechanism. The system sends heartbeat packets to each Pool within a set period and waits for a response. When the response is not received continuously for multiple times (such as 3 times or 5 times), it can be inferred that there is a risk of network unreachability for the Pool. The preset number of times should not be hard-coded as a fixed value, but should be configured as an adjustable parameter to facilitate subsequent flexible optimization according to business stability and network environment.

[0187] The performance exception status is implemented by the system's dynamic analysis of the response time. The average response time of each Pool is an important indicator reflecting the service quality and will be maintained within a certain range under stable operating conditions. By comparing its ratio with the preset time threshold, for example, exceeding 2 times or 3 times, the system can consider that the Pool has entered the performance degradation state. In such cases, to avoid affecting high-concurrency write requests, the new write task allocation for it should be suspended immediately.

[0188] In the actual processing process, if the same Pool meets multiple exception determination conditions simultaneously, the system will mark all exception types in parallel. The marked information includes fields such as exception type, trigger time, severity level, etc., which serve as the basis for subsequent recovery and analysis. To ensure business continuity and system stability, the hardware exception status is given the highest priority, and the priority determination logic can be supported by the exception type dictionary to ensure system consistency.

[0189] The action of interrupting the write request is executed immediately after the exception status is determined. It mainly updates the write channel status of the scheduling module, sets the scheduling pointer of the Pool to be invalidated, and at the same time sets its current write weight to zero. Before performing the weight adjustment, the current write weight value is backed up to the historical weight record table to retain its load level in the last normal state. This historical record facilitates the weight rollback operation or operation and maintenance analysis after subsequent exception recovery. The table can include fields such as timestamp, exception type, and original weight value.

[0190] This embodiment can effectively prevent the spread of hardware failures, chain write failures caused by network anomalies, and overall system write latency caused by performance bottlenecks by performing fine-grained determination and classification of the exception status of the storage Pool and implementing a priority isolation strategy based on the exception type. Further, by setting the weight to zero and interrupting the write request, it avoids continuing to import data into the faulty Pool, ensuring data security and write high availability. At the same time, the backup mechanism of the historical weight record improves the system's adaptability to fault recovery and reduces the manual tuning burden.

[0191] In one embodiment, after the above step 70 and before step S80, it further includes:

[0192] S701, start a decay timer for the Pool whose write weight has been reduced during the current monitoring period, and record the weight reduction timestamp of the Pool whose write weight has been reduced;

[0193] S701, in each subsequent monitoring period, detect whether the available storage space of the Pool whose write weight has been reduced is continuously higher than the preset space threshold and whether the average response time of the Pool whose write weight has been reduced is lower than the preset time threshold;

[0194] S701, if the available storage space of the Pool with the reduced write weight has been continuously higher than the preset space threshold and the average response time has been lower than the preset time threshold within multiple consecutive monitoring periods recorded by the decay timer, then gradually increase the current write weight of the Pool with the reduced write weight at a preset recovery rate until the current write weight of the Pool with the reduced write weight is restored to the initial reference weight;

[0195] S701, if the write weight of a certain Pool is increased due to the data access frequency exceeding the preset frequency threshold during the current monitoring period, start the frequency stability detection and record the weight increase timestamp of the Pool whose data access frequency exceeds the preset frequency threshold;

[0196] S701, in each subsequent monitoring period, detect whether the data access frequency of the Pool whose data access frequency exceeds the preset frequency threshold remains higher than a preset proportion of the preset frequency threshold;

[0197] S701, if the data access frequency of the Pool whose data access frequency exceeds the preset frequency threshold remains higher than a preset proportion of the preset frequency threshold within multiple consecutive time windows recorded by the frequency stability detection, then lock the current write weight of the Pool whose data access frequency exceeds the preset frequency threshold as the current maximum value and prohibit subsequent weight reduction operations on the Pool whose data access frequency exceeds the preset frequency threshold;

[0198] S701, for the abnormal Pool whose write weight is set to zero during the current monitoring period, send a standardized stress test data packet to the abnormal Pool whose write weight is set to zero after the fault is repaired;

[0199] S701, based on the execution result of the standardized stress test data packet, detect whether the read / write input / output rate of the abnormal Pool whose write weight is set to zero has been restored to the global average peak rate and whether the response time volatility of the abnormal Pool whose write weight is set to zero meets the preset stability condition;

[0200] S701, if the read / write input / output rate of the abnormal Pool whose write weight is set to zero has been restored to the global average peak rate and the response time volatility meets the preset stability condition, then reactivate the abnormal Pool whose write weight is set to zero in the next monitoring period and initialize the current write weight of the abnormal Pool whose write weight is set to zero to a preset safety value;

[0201] S701, record the number of weight adjustments, the adjustment range, and the change rate of the performance metrics after adjustment for each Pool, generate weight sensitivity parameters based on time series analysis according to the number of weight adjustments, the adjustment range, and the change rate of the performance metrics after adjustment, and feedback the weight sensitivity parameters to the analysis module with a preset space threshold and a preset rate benchmark, and adjust the initial setting values of the preset space threshold and the preset rate benchmark in subsequent monitoring cycles according to a preset correction strategy.

[0202] In this embodiment, after converting the current write weight of each Pool into a probability value and constructing a discrete probability distribution table based on this probability value, in order to improve the accuracy of weight calculation and the resource allocation efficiency in subsequent monitoring cycles, the system executes a set of operations for weight status tracking and feedback optimization. These operations do not affect the result of using the probability distribution in the current cycle, but provide a basis for judging the subsequent weight change trend.

[0203] First, for the Pool whose write weight is reduced in the current monitoring cycle, start a decay timer and record the corresponding weight reduction timestamp, which is used to accurately calibrate the start time of this weight adjustment. The basis of this operation is to judge whether this Pool has the basic conditions to restore its weight in the future and avoid being in a disadvantaged allocation state for a long time.

[0204] Subsequently, in each subsequent monitoring cycle, the system continuously detects the resource usage status of the Pool whose write weight is reduced, including whether its available storage space is continuously higher than the system preset space threshold and whether the average response time is continuously lower than the preset time threshold. If these two indicators always meet the requirements in multiple consecutive cycles, it is considered that this Pool has restored the resource health state. The system gradually restores its current write weight according to the preset recovery rate, and the recovery upper limit is the initial reference weight to avoid a sudden increase in load caused by a one-time recovery.

[0205] Secondly, for the Pool whose write weight is increased due to the increase in access frequency in the current monitoring cycle, the system starts a frequency stability detection process and records its weight increase timestamp. In each subsequent monitoring cycle, the system judges whether the data access frequency of this Pool is stably higher than the ratio standard of the preset frequency threshold in multiple consecutive time windows. If this stability condition is met, it is considered that this Pool belongs to stable hot resources, and the system locks its current maximum write weight and prohibits subsequent weight reduction operations caused by condition changes, so as to ensure the write stability of hot resources.

[0206] Meanwhile, to handle the abnormal Pool with zeroed weights in the current cycle, after the fault is marked as repaired, the system sends a standardized stress test data packet to it, and the test content includes simulating the standard read-write request load. The system analyzes the performance recovery status of this Pool based on the data returned by the test, mainly including whether its read-write input-output rate has recovered to the system-level global average peak rate, and whether the response time volatility is less than the preset stability threshold. If both indicators meet the standards, the system reactivates the write ability of this Pool in the next monitoring cycle and initializes its current write weight to a preset safety value to ensure the overall stability of the system when it resumes participating in scheduling.

[0207] Finally, to improve the adaptive ability of future weight adjustment and threshold setting, the system records the historical adjustment data for each Pool, including the number of times, amplitude of each weight adjustment, and the changes in relevant performance indicators (such as response time, read-write rate) after the adjustment. Based on these time series data, the system conducts a weight sensitivity analysis to calculate the response intensity of each Pool to different parameter changes. The analysis results are fed back to the threshold setting module in the form of weight sensitivity parameters to dynamically correct the initial values of parameters such as the preset space threshold and the preset rate benchmark, thereby realizing the closed-loop optimization of threshold adjustment.

[0208] Example illustration: In the urban intelligent medical center, the data management system executes the multi-Pool writing strategy with a minute as the cycle. During a certain night shift period, PoolA was down-weighted due to the storage space approaching the threshold and the response time increasing in the current monitoring cycle. The system immediately records the timestamp of the weight reduction and starts the decay timer. However, the discrete probability distribution table for the current cycle has been generated based on this down-weighted state, so PoolA will not be selected as the target Pool in the current cycle. The system continuously monitors the operating state of PoolA in several subsequent cycles and detects that its available space continues to recover and the response time is stably lower than the set value. After meeting the conditions, the system gradually increases its writing weight according to the recovery rate in the sixth monitoring cycle, enabling it to regain the ability to participate in scheduling in subsequent cycles, thus alleviating the writing bottleneck of imaging data during the night shift. At the same time, PoolB on the ICU monitoring platform has its writing weight immediately increased and the triggering time recorded because the vital sign data of patients has been frequently accessed for several consecutive days in the current cycle, and the access frequency exceeds the set threshold. Although this adjustment has been used in the probability distribution table for this cycle, the system also starts the frequency stability detection logic and observes whether it maintains the high-frequency access state in subsequent cycles. After seven consecutive time window detections, the access frequency of PoolB always remains above the preset ratio. Based on this, the system locks its weight at the maximum value in the eighth cycle and prohibits automatic down-regulation in subsequent writing strategies to ensure the stability of high-intensity data query tasks. Another PoolC used for caching intermediate results of blood image analysis is detected to have an SSD wear degree exceeding the health threshold in the current cycle, triggering an abnormal state and immediately being set to zero weight. After excluding the writing path, the system sends a standardized stress test data packet for performance verification. The test results show that the read and write rates of this Pool have recovered to the global average peak level after reconstruction, and the response time fluctuation is controlled within a stable range. Therefore, the system arranges to restore its writing weight to the preset safety value in the next monitoring cycle and re-incorporates it into the scheduling path to relieve the overload pressure on other Pools. The background scheduling module continuously archives the weight adjustment trajectories of all Pools, such as the length of the down-weighting and recovery cycle of PoolA, the length of the frequency locking window of PoolB, the number of abnormal repairs of PoolC, etc. The system summarizes the data after each cycle, executes the training of the time series analysis model, obtains the response sensitivity of the Pool to threshold changes, and feeds it back to the subsequent threshold calculation module. Based on this, the scheduling module fine-tunes the frequency threshold of PoolB and the response time upper limit of PoolC to ensure that the system can be more adaptive and accurately allocate resources in the future under high-intensity medical pressure.

[0209] In the real-time matching trading system of a large securities firm, PoolX, which is used for order flow buffering, monitors that the IOPS in the current trading cycle is higher than the historical average, and at the same time, the response time is slightly delayed, triggering the downgrading mechanism. The system immediately records the time point when the weight is reduced and generates the discrete probability distribution table for the current cycle, but does not perform immediate repair. In the subsequent cycles, the system detects that the available space and response delay of PoolX continue to recover. In the 5th cycle, the system adjusts its write weight according to the preset recovery rate, enabling it to gradually regain the matching buffer right and improve the write channel elasticity during the opening stage of the next day. During the peak season of quarterly report disclosure, the query requests for announcement documents carried by PoolY surge, and the access frequency in the current cycle exceeds the threshold. The system records this status and increases the write weight for the selection of the target Pool in the current cycle. At the same time, the system starts the frequency stability detection and enters the subsequent cycles for access density tracking. After 9 consecutive cycles of detection, PoolY always maintains a high-frequency read request. The system determines it as an access hot spot, locks its weight, and prohibits any downward adjustment in the subsequent trading cycles to ensure uninterrupted information reading. During the overnight batch processing, PoolZ, which carries financial logs, is determined as an abnormal Pool because it continuously loses heartbeat signals due to network link jitter, and its weight is set to zero. In the current cycle, it is excluded from the scheduling path, and a stress test data packet is sent to it. The detection shows that its performance recovers well after reconnection. The system reactivates PoolZ in the next cycle and sets its initial write weight to the system security reference value to ensure the multi-path redundancy recovery of the log write link. At the same time, the system records the weight adjustment records of PoolX / Y / Z in this cycle, and detects that PoolX is extremely sensitive to response time fluctuations, while PoolY is highly sticky to access frequency and is suitable for a higher threshold tolerance. The system obtains the weight adjustment sensitivity of each Pool through time series learning and feeds it back to optimize the space threshold, frequency threshold, and weight upper limit strategies in the next cycle, realizing the precision-level scheduling of the resources of the matching system.

[0210] Through the continuous tracking of the write weight status and the delayed feedback mechanism in this embodiment, the system not only realizes the linkage management of the current resource usage status and historical performance changes, but also establishes a threshold self-adaptive optimization mechanism through structured time series analysis. It enables the write weight to no longer be limited to real-time judgment, but to have the ability of trend prediction and correction. In the face of complex operation scenarios such as high-frequency access, resource recovery, or abnormal repair, the system can achieve precise control of the write scheduling strategy through fine trigger conditions and recovery criteria, improving the intelligent level of write scheduling and the overall resource usage efficiency.

[0211] In one embodiment, the above step S80 includes:

[0212] S801. Based on the normalized probability values of each Pool in the discrete probability distribution table, construct probability intervals corresponding to the Pools one by one. The probability intervals cover a preset numerical range, and the interval lengths of each Pool are proportional to the corresponding normalized probability values.

[0213] S802. Call a pseudo-random number generator to generate a random number within the preset numerical range.

[0214] S803. Traverse the probability intervals of all Pools in a preset order, and determine whether the random number falls within the interval range of the currently traversed Pool.

[0215] S804. If the random number falls within the interval range of the currently traversed Pool, terminate the traversal and select the current Pool as the target Pool.

[0216] S805. If the random number does not fall within the interval range of any Pool, determine the target Pool according to a preset fault tolerance strategy.

[0217] In this embodiment, the determination mechanism of the target Pool is centered around discrete probability modeling, probability interval mapping, and pseudo-random sampling. Essentially, it transforms the continuous resource scheduling problem into a fast-executable interval landing point determination problem under the constraint of limited computing resources. The core lies in taking the current write weight of the Pool as the input, constructing a discrete probability space through normalization, and introducing an efficient pseudo-random sampling method to achieve the unity of dynamics, flexibility, and fairness in target Pool selection.

[0218] First, the normalized weight refers to dividing the write weight value of each Pool by the sum of the write weight values of all Pools to generate a normalized probability value in the range of [0, 1]. This processing ensures the relative proportion of each Pool in the total weight and reflects its expected frequency of being "selected". This normalization process originates from the classical weighted sampling model and is widely used in fields such as scheduling algorithms, image pixel sampling, and the neural network Dropout mechanism.

[0219] To facilitate the landing point judgment, the normalized probability values need to be mapped into a series of non-overlapping intervals to construct a discrete probability distribution that covers the continuous numerical space (such as [0, 1]). Each Pool corresponds to an interval, and the interval length is equal to its normalized probability value. For example, the interval of the first Pool is [0, P1), the second is [P1, P1 + P2), and so on until the last Pool, ensuring that the intervals are seamlessly connected, non-overlapping, and without omission. This interval division is equivalent to constructing a one-dimensional cumulative probability density function (CDF), which is the mathematical basis behind most random selectors (such as the roulette wheel algorithm).

[0220] The pseudo-random number generator is used to simulate a sampling operation, which generates a random value with an approximately equal probability distribution within a range. This generator can be implemented based on the software-based linear congruential method (LCG), Mersenne Twister, or by calling the hardware entropy source provided at the system level (such as / dev / random, RDRAND instruction). Its quality has a direct impact on the fairness of target selection and the system convergence. Especially in a distributed writing system that requires frequent sampling, the periodicity and repeatability of pseudo-random numbers must be controlled.

[0221] The judgment process adopts a sequential traversal or binary search strategy. The former is simple to implement and suitable for scenarios with a limited number of Pools; the latter is suitable for optimizing the search performance when the number of Pools is large. By determining whether a random number falls into a certain probability interval, the target Pool can be quickly determined. This step has a constant-level complexity, ensuring the system throughput capacity in high-frequency writing scenarios.

[0222] Due to extreme situations such as floating-point accumulation errors or sudden weight fluctuations, it is possible that a random number does not hit any interval. Therefore, a fault-tolerant processing path needs to be designed. In the default mode, the Pool with the highest current write weight can be selected as the fallback target; the least used Pool can also be selected based on historical hit frequencies to achieve reverse heat adjustment, or a default backup Pool can be set in combination with business priorities to enhance the robustness of the mechanism.

[0223] This process can not only map the scheduling strategy changes brought about by the changes in write weights, but also avoid the polarization problem caused by the traditional "highest weight first" (i.e., resources always tend to a certain Pool). By introducing "biased randomness", it improves the global resource utilization rate and local performance balance ability of the entire system. In addition, the pseudo-random mechanism also has the natural characteristic of tolerating asynchronous fluctuations, making it highly adaptable in complex environments such as distributed systems, edge nodes, and multi-cluster coordination.

[0224] In this embodiment, by constructing a discrete probability distribution table dynamically associated with the Pool write weights and combining the pseudo-random number mechanism for target Pool selection, elastic scheduling and adaptive balance of the write path in a high-load environment are achieved. Without increasing the system burden, the randomness and fairness of write target selection are improved, and long-term resource skew is avoided. Combined with the fault-tolerant strategy, it can also effectively prevent write failure problems caused by random anomalies or precision errors, enhancing the robustness and disaster tolerance of the system.

[0225] In one embodiment, a multi-Pool storage resource adaptation write device is provided, and this multi-Pool storage resource adaptation write device corresponds one-to-one with the multi-Pool storage resource adaptation write method in the above embodiment. Refer to Figure 3 , Figure 3This is a schematic diagram of the functional modules of a preferred embodiment of the multi-Pool storage resource adaptation writing device of the present invention. The monitoring module 10, the space regulation module 20, the rate regulation module 30, the frequency regulation module 40, the response regulation module 50, the exception handling module 60, the probability construction module 70, the target selection module 80, and the writing execution module 90. The detailed description of each functional module is as follows:

[0226] The monitoring module 10 is used to monitor the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool;

[0227] The space regulation module 20 is used to reduce the current writing weight of the Pool whose available storage space is lower than the preset space threshold when it is detected that there is a Pool with available storage space lower than the preset space threshold;

[0228] The rate regulation module 30 is used to increase the current writing weight of the Pool whose read / write input / output rate is higher than the preset rate benchmark when it is detected that there is a Pool with read / write input / output rate higher than the preset rate benchmark;

[0229] The frequency regulation module 40 is used to increase the current writing weight of the Pool whose data access frequency exceeds the preset frequency threshold when it is detected that there is a Pool with data access frequency exceeding the preset frequency threshold;

[0230] The response regulation module 50 is used to reduce the current writing weight of the Pool whose average response time exceeds the preset time threshold when it is detected that there is a Pool with average response time exceeding the preset time threshold;

[0231] The exception handling module 60 is used to set the current writing weight of the Pool in an abnormal state to zero if there is a Pool in an abnormal state;

[0232] The probability construction module 70 is used to convert the current writing weight of each Pool into a probability value and construct a discrete probability distribution table based on the probability value;

[0233] The target selection module 80 is used to determine the target Pool according to the discrete probability distribution table through a pseudo-random number generator;

[0234] The writing execution module 90 is used to write the data to be stored into the target Pool.

[0235] In an embodiment, the monitoring module 10 is specifically used for:

[0236] Perform millisecond-level periodic scans on the storage devices of each Pool to generate available storage space values;

[0237] Based on the sliding window statistical strategy, count the total number of read and write operations of each Pool within a preset time window to generate read and write input / output rate values;

[0238] Measure the end-to-end response time of the data requests of each Pool, and deduct the preset network transmission benchmark delay value to generate a corrected average response time value;

[0239] Classify and count the data access times of each Pool per unit time according to user information, transaction records, and log data types to generate data access frequency values.

[0240] In one embodiment, the monitoring module 10 is specifically configured to:

[0241] Obtain the total storage capacity of all Pools, and count the average historical storage growth rate of each Pool;

[0242] Based on the total storage capacity and the average historical storage growth rate, determine the predicted value of the global storage occupancy rate for a future preset period through a linear extrapolation module;

[0243] Set the preset space threshold as the difference percentage between the total storage capacity and the predicted value of the global storage occupancy rate;

[0244] Obtain the peak data of the historical read and write input / output rates of all Pools, and determine the global average peak rate based on the peak data of the historical read and write input / output rates;

[0245] Monitor the comprehensive load rate of the current system, and adjust the global average peak rate according to the comprehensive load rate to generate the preset rate benchmark.

[0246] In one embodiment, the monitoring module 10 is specifically configured to:

[0247] Access the event calendar data of the external service module, and parse the event type code and time interval in the event calendar data;

[0248] Filter out the events with the event type code being the preset high-concurrency event type, and use the time interval of the corresponding events as the system-level peak period;

[0249] Compare the current time with the start and end times of the system-level peak period;

[0250] If the current time is within the time interval of any system-level peak period, raise the preset frequency threshold to the preset emergency threshold;

[0251] If the current time is outside the time intervals of all system-level peak periods, restore the preset frequency threshold to the preset normal threshold.

[0252] In one embodiment, the monitoring module 10 is specifically configured to:

[0253] Collect health status indicators of all storage devices bound to the Pool, including disk SMART status parameters, solid-state drive wear percentage, and disk array degradation event count;

[0254] Perform a standardized scoring on the health status indicators of each storage device to generate a health score for each storage device;

[0255] Determine the mean of the health scores of all storage devices and use the mean as the global hardware health score;

[0256] When the initialization of the storage device is completed or within a preset calibration period, send a standardized read / write test instruction to all storage devices bound to the Pool;

[0257] Measure the time interval from the issuance of the standardized read / write test instruction to the storage device's return of an operation completion confirmation signal as the reference response time of a single storage device;

[0258] Summarize the reference response times of all storage devices and use the average of the reference response times of all storage devices as the global reference response time mean;

[0259] Based on the global hardware health score and the global reference response time mean, determine a redundancy time compensation value;

[0260] Set a preset time threshold as the sum of the reference response time mean and the redundancy time compensation value.

[0261] In one embodiment, the exception handling module 60 is specifically configured to:

[0262] When it is detected that a hardware failure signal is triggered by a storage device bound to the Pool, it is determined that the Pool is in a hardware exception state;

[0263] When it is detected that the Pool continuously loses heartbeat signals a preset number of times, it is determined that the Pool is in a network exception state;

[0264] When it is detected that the average response time of the Pool exceeds a preset multiple of the preset time threshold, it is determined that the Pool is in a performance exception state;

[0265] Interrupt the reception of write requests for the Pool in an exception state, set the current write weight of the Pool in an exception state to zero, and mark the corresponding exception state type.

[0266] In one embodiment, the probability construction module 70 is specifically configured to:

[0267] Start a decay timer for the Pool whose write weight has been reduced during the current monitoring period, and record the weight reduction timestamp of the Pool whose write weight has been reduced.

[0268] In each subsequent monitoring period, detect whether the available storage space of the Pool whose write weight has been reduced continues to be higher than the preset space threshold and whether the average response time of the Pool whose write weight has been reduced is lower than the preset time threshold.

[0269] If the available storage space of the Pool whose write weight has been reduced continues to be higher than the preset space threshold and the average response time is lower than the preset time threshold in multiple consecutive monitoring periods recorded by the decay timer, gradually increase the current write weight of the Pool whose write weight has been reduced at a preset recovery rate until the current write weight of the Pool whose write weight has been reduced is restored to the initial reference weight.

[0270] If the write weight of a certain Pool is increased due to the data access frequency exceeding the preset frequency threshold during the current monitoring period, start frequency stability detection and record the weight increase timestamp of the Pool whose data access frequency exceeds the preset frequency threshold.

[0271] In each subsequent monitoring period, detect whether the data access frequency of the Pool whose data access frequency exceeds the preset frequency threshold remains higher than a preset percentage of the preset frequency threshold.

[0272] If the data access frequency of the Pool whose data access frequency exceeds the preset frequency threshold remains higher than a preset percentage of the preset frequency threshold in multiple consecutive time windows recorded by the frequency stability detection, lock the current write weight of the Pool whose data access frequency exceeds the preset frequency threshold to the current maximum value, and prohibit subsequent weight reduction operations on the Pool whose data access frequency exceeds the preset frequency threshold.

[0273] For the abnormal Pool whose write weight has been set to zero during the current monitoring period, send a standardized stress test data packet to the abnormal Pool whose write weight has been set to zero after the fault is repaired.

[0274] Based on the execution result of the standardized stress test data packet, detect whether the read / write input / output rate of the abnormal Pool whose write weight has been set to zero has recovered to the global average peak rate and whether the response time volatility of the abnormal Pool whose write weight has been set to zero meets the preset stability condition.

[0275] If the read / write input / output rate of the abnormal Pool with the write weight set to zero resumes to the global average peak rate and the response time volatility meets the preset stability condition, then in the next monitoring period, reactivate the abnormal Pool with the write weight set to zero and initialize the current write weight of the abnormal Pool with the write weight set to zero to a preset safety value;

[0276] Record the weight adjustment times, adjustment amplitudes, and change rates of the adjusted performance metrics of each Pool, generate weight sensitivity parameters based on time series analysis according to the weight adjustment times, adjustment amplitudes, and change rates of the adjusted performance metrics, and feedback the weight sensitivity parameters to the analysis module of the preset space threshold and preset rate benchmark, and adjust the initial setting values of the preset space threshold and preset rate benchmark in the subsequent monitoring period according to the preset correction strategy.

[0277] In one embodiment, the target selection module 80 is specifically configured to:

[0278] Based on the normalized probability values of each Pool in the discrete probability distribution table, construct a probability interval corresponding to each Pool one by one. The probability interval covers a preset numerical range, and the interval length of each Pool is proportional to the corresponding normalized probability value;

[0279] Call a pseudo-random number generator to generate a random number within the preset numerical range;

[0280] Traverse the probability intervals of all Pools in a preset order, and determine whether the random number falls within the interval range of the currently traversed Pool;

[0281] If the random number falls within the interval range of the currently traversed Pool, terminate the traversal and select the current Pool as the target Pool;

[0282] If the random number does not fall within the interval range of any Pool, determine the target Pool according to the preset fault tolerance strategy.

[0283] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 4As shown in the figure. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a multi-Pool storage resource adaptation writing method.

[0284] In one embodiment, a computer device is provided. The computer device can be a client, and its internal structure diagram can be as Figure 5 shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the client side of a multi-Pool storage resource adaptation writing method

[0285] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are realized:

[0286] Monitor the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool;

[0287] When it is monitored that the available storage space of a Pool is lower than a preset space threshold, reduce the current writing weight of the Pool whose available storage space is lower than the preset space threshold;

[0288] When it is monitored that the read / write input / output rate of a Pool is higher than a preset rate benchmark, increase the current writing weight of the Pool whose read / write input / output rate is higher than the preset rate benchmark;

[0289] When it is monitored that the data access frequency of a Pool exceeds a preset frequency threshold, increase the current writing weight of the Pool whose data access frequency exceeds the preset frequency threshold;

[0290] When it is monitored that the average response time of a Pool exceeds a preset time threshold, the current write weight of the Pool whose average response time exceeds the preset time threshold is reduced;

[0291] If there is a Pool in an abnormal state, the current write weight of the Pool in the abnormal state is set to zero;

[0292] Convert the current write weight of each Pool into a probability value, and construct a discrete probability distribution table based on the probability value;

[0293] Determine the target Pool through a pseudo-random number generator according to the discrete probability distribution table;

[0294] Write the data to be stored into the target Pool.

[0295] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0296] Monitor the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool;

[0297] When it is monitored that the available storage space of a Pool is lower than a preset space threshold, the current write weight of the Pool whose available storage space is lower than the preset space threshold is reduced;

[0298] When it is monitored that the read / write input / output rate of a Pool is higher than a preset rate benchmark, the current write weight of the Pool whose read / write input / output rate is higher than the preset rate benchmark is increased;

[0299] When it is monitored that the data access frequency of a Pool exceeds a preset frequency threshold, the current write weight of the Pool whose data access frequency exceeds the preset frequency threshold is increased;

[0300] When it is monitored that the average response time of a Pool exceeds a preset time threshold, the current write weight of the Pool whose average response time exceeds the preset time threshold is reduced;

[0301] If there is a Pool in an abnormal state, the current write weight of the Pool in the abnormal state is set to zero;

[0302] Convert the current write weight of each Pool into a probability value, and construct a discrete probability distribution table based on the probability value;

[0303] Determine the target Pool through a pseudo-random number generator according to the discrete probability distribution table;

[0304] Write the data to be stored into the target Pool.

[0305] It should be noted that for the functions or steps that can be achieved by the above-mentioned computer-readable storage medium or computer device, reference can be made to the relevant descriptions on the server side and the user side in the foregoing method embodiments. To avoid repetition, they will not be described in detail here.

[0306] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned method embodiments. Among them, any reference to the memory, storage, database or other media used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0307] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0308] It should be noted that if there are software tools or components of other companies in the embodiments of the present application, they are only used for example introduction and do not represent actual use. The above-mentioned embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A multi-Pool storage resource adaptation writing method, characterized in that, Including the following steps: Monitoring the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool; When it is monitored that the available storage space of a Pool is lower than the preset space threshold, reducing the current write weight of the Pool whose available storage space is lower than the preset space threshold; When it is monitored that the read / write input / output rate of a Pool is higher than the preset rate benchmark, increasing the current write weight of the Pool whose read / write input / output rate is higher than the preset rate benchmark; When it is monitored that the data access frequency of a Pool exceeds the preset frequency threshold, increasing the current write weight of the Pool whose data access frequency exceeds the preset frequency threshold; When it is monitored that the average response time of a Pool exceeds the preset time threshold, reducing the current write weight of the Pool whose average response time exceeds the preset time threshold; If there is a Pool in an abnormal state, setting the current write weight of the Pool in the abnormal state to zero; Converting the current write weight of each Pool into a probability value and constructing a discrete probability distribution table based on the probability value; Determining the target Pool through a pseudo-random number generator according to the discrete probability distribution table; Writing the data to be stored into the target Pool.

2. The multi-Pool storage resource adaptation writing method according to claim 1, wherein Monitoring the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool includes: Performing a millisecond-level periodic scan on the storage device of each Pool to generate the available storage space value; Based on the sliding window statistical strategy, counting the total number of read / write operations of each Pool within a preset time window to generate the read / write input / output rate value; Measuring the end-to-end response time of the data request of each Pool and deducting the preset network transmission benchmark delay value to generate the corrected average response time value; Classifying and counting the data access times of each Pool per unit time according to user information, transaction records, and log data types to generate the data access frequency value.

3. The multi-Pool storage resource adaptation writing method according to claim 1, wherein, After monitoring the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool and before it is monitored that the read / write input / output rate of a Pool is higher than the preset rate benchmark, it further includes: Obtaining the total storage capacity of all Pools and counting the average value of the historical storage growth rate of each Pool; Based on the total storage capacity and the average value of the historical storage growth rate, determining the predicted value of the global storage occupancy rate for a future preset period through a linear extrapolation module; Setting the preset space threshold as the difference percentage between the total storage capacity and the predicted value of the global storage occupancy rate; Obtaining the peak data of the historical read / write input / output rate of all Pools and determining the global average peak rate based on the peak data of the historical read / write input / output rate; Monitoring the comprehensive load rate of the current system and adjusting the global average peak rate according to the comprehensive load rate to generate the preset rate benchmark.

4. The multi-Pool storage resource adaptation writing method according to claim 1, characterized in that After monitoring the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool, and before it is detected that the read / write input / output rate of a Pool is higher than the preset rate benchmark, it further includes: Access the event calendar data of the external service module and parse the event type code and time interval in the event calendar data; Filter the events with the event type code being the preset high-concurrency event type, and use the time interval of the corresponding events as the system-level peak period; Compare the current time with the start and end times of the system-level peak period; If the current time is within the time interval of any system-level peak period, increase the preset frequency threshold to the preset emergency threshold; If the current time is outside the time intervals of all system-level peak periods, restore the preset frequency threshold to the preset normal threshold.

5. The multi-Pool storage resource adaptation writing method according to claim 1, wherein After monitoring the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool, and before it is detected that the average response time of a Pool exceeds the preset time threshold, it further includes: Collect the health status indicators of the storage devices bound to all Pools, including disk SMART status parameters, solid-state drive wear percentage, and disk array degradation event count; Perform a standardized scoring on the health status indicators of each storage device to generate the health score of each storage device; Determine the mean of the health scores of all storage devices and use the mean as the global hardware health score; When the initialization of the storage device is completed or within the preset calibration period, send a standardized read / write test instruction to the storage devices bound to all Pools; Measure the time interval from the issuance of the standardized read / write test instruction to the return of the operation completion confirmation signal by the storage device as the reference response time of a single storage device; Summarize the reference response times of all storage devices and use the average of the reference response times of all storage devices as the global reference response time mean; Based on the global hardware health score and the global reference response time mean, determine the redundant time compensation value; Set the preset time threshold to the sum of the reference response time mean and the redundant time compensation value.

6. The multi-Pool storage resource adaptation write method according to claim 1, characterized in that, After converting the current write weight of each Pool into a probability value and constructing a discrete probability distribution table based on the probability value, and before determining the target Pool through a pseudo-random number generator according to the discrete probability distribution table, it further includes: Start a decay timer for the Pool whose write weight has been reduced during the current monitoring period, and record the weight reduction timestamp of the Pool whose write weight has been reduced; In each subsequent monitoring period, detect whether the available storage space of the Pool whose write weight has been reduced is continuously higher than the preset space threshold and whether the average response time of the Pool whose write weight has been reduced is lower than the preset time threshold; If the available storage space of the Pool with the reduced write weight continuously exceeds the preset space threshold and the average response time is lower than the preset time threshold within a continuous number of monitoring cycles recorded by the decay timer, the current write weight of the Pool with the reduced write weight is gradually increased at a preset recovery rate until the current write weight of the Pool with the reduced write weight is restored to the initial reference weight; If the write weight of a certain Pool is increased due to the data access frequency exceeding the preset frequency threshold during the current monitoring cycle, start the frequency stability detection and record the weight increase timestamp of the Pool whose data access frequency exceeds the preset frequency threshold; In each subsequent monitoring cycle, detect whether the data access frequency of the Pool whose data access frequency exceeds the preset frequency threshold remains higher than a preset percentage of the preset frequency threshold; If the data access frequency of the Pool whose data access frequency exceeds the preset frequency threshold remains higher than a preset percentage of the preset frequency threshold within a continuous number of time windows recorded by the frequency stability detection, lock the current write weight of the Pool whose data access frequency exceeds the preset frequency threshold to the current maximum value and prohibit subsequent weight reduction operations on the Pool whose data access frequency exceeds the preset frequency threshold; For the abnormal Pool with the write weight set to zero during the current monitoring cycle, send a standardized stress test data packet to the abnormal Pool with the write weight set to zero after the fault is repaired; Based on the execution result of the standardized stress test data packet, detect whether the read / write input / output rate of the abnormal Pool with the write weight set to zero has recovered to the global average peak rate and whether the response time volatility of the abnormal Pool with the write weight set to zero meets the preset stability condition; If the read / write input / output rate of the abnormal Pool with the write weight set to zero has recovered to the global average peak rate and the response time volatility meets the preset stability condition, reactivate the abnormal Pool with the write weight set to zero in the next monitoring cycle and initialize the current write weight of the abnormal Pool with the write weight set to zero to the preset safety value; Record the number of weight adjustments, adjustment amplitude, and change rate of the performance index after adjustment for each Pool, and generate a weight sensitivity parameter based on time series analysis according to the number of weight adjustments, adjustment amplitude, and change rate of the performance index after adjustment. Feed back the weight sensitivity parameter to the analysis module of the preset space threshold and preset rate benchmark, and adjust the initial setting values of the preset space threshold and preset rate benchmark in the subsequent monitoring cycle according to the preset correction strategy.

7. The multi-Pool storage resource adaptation write method according to claim 1, characterized in that, According to the discrete probability distribution table, determine the target Pool through a pseudo-random number generator, including: Based on the normalized probability values of each Pool in the discrete probability distribution table, construct a probability interval corresponding to each Pool one by one. The probability interval covers the preset numerical range, and the interval length of each Pool is proportional to the corresponding normalized probability value; Call the pseudo-random number generator to generate a random number within the preset numerical range; Traverse the probability intervals of all Pools according to a preset order, and determine whether the random number falls within the interval range of the currently traversed Pool; If the random number falls within the interval range of the currently traversed Pool, terminate the traversal and select the current Pool as the target Pool; If the random number does not fall within the interval range of any Pool, determine the target Pool according to a preset fault tolerance strategy.

8. A multi-Pool storage resource adaptation writing device, characterized in that, The multi-Pool storage resource adaptation writing device includes: A monitoring module, configured to monitor the available storage space, read / write input / output rate, average response time, and data access frequency of each Pool; A space regulation module, configured to reduce the current writing weight of the Pool whose available storage space is lower than a preset space threshold when it is detected that there is a Pool whose available storage space is lower than the preset space threshold; A rate regulation module, configured to increase the current writing weight of the Pool whose read / write input / output rate is higher than a preset rate benchmark when it is detected that there is a Pool whose read / write input / output rate is higher than the preset rate benchmark; A frequency regulation module, configured to increase the current writing weight of the Pool whose data access frequency exceeds a preset frequency threshold when it is detected that there is a Pool whose data access frequency exceeds the preset frequency threshold; A response regulation module, configured to reduce the current writing weight of the Pool whose average response time exceeds a preset time threshold when it is detected that there is a Pool whose average response time exceeds the preset time threshold; An exception handling module, configured to set the current writing weight of the Pool in an abnormal state to zero if there is a Pool in an abnormal state; A probability construction module, configured to convert the current writing weight of each Pool into a probability value, and construct a discrete probability distribution table based on the probability value; A target selection module, configured to determine the target Pool through a pseudo-random number generator according to the discrete probability distribution table; A writing execution module, configured to write the data to be stored into the target Pool.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a multi-Pool storage resource adaptation writing program stored in the memory and executable on the processor. When the multi-Pool storage resource adaptation writing program is executed by the processor, the steps of the multi-Pool storage resource adaptation writing method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The storage medium stores a multi-Pool storage resource adaptation writing program, and when the multi-Pool storage resource adaptation writing program is executed by a processor, the steps of the multi-Pool storage resource adaptation writing method according to any one of claims 1-7 are implemented.

Images