Load awareness and data dynamic layout method and system based on deep learning

Abstract

The invention relates to the technical field of data processing, in particular to a load awareness and data dynamic layout method and system based on deep learning, and the method comprises the steps: collecting hardware load indexes of all nodes in a distributed cluster in real time, and carrying out the preprocessing of the hardware load indexes; based on the preprocessed hardware load index, calculating a comprehensive load score of each node through a weighted fuzzy logic algorithm; dividing load intervals according to the comprehensive load score and matching a predefined strategy combination; and dynamically adjusting a compression algorithm, a compression ratio and a fragment merging parameter based on the current service flow peak-valley state of the target data block and a matched predefined strategy combination, the service flow peak-valley state being determined through a request processing rate and response time. The objective of the invention is to realize dynamic and fine-grained load sensing and data layout.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a load perception and data dynamic layout method and system based on deep learning. Background Art

[0002] In the computer storage field, existing database systems generally adopt a static storage model design. Core parameters such as the compression algorithm, compression ratio, row-column storage mode, and fragment merging trigger conditions are fixed during system initialization. For example, typical solutions default to a fixed compression algorithm such as LZ4 or Snappy, with a uniform compression ratio of 4:1. Fragment merging is triggered by a fixed time interval or fragment size threshold.

[0003] Because server resource configuration is usually statically allocated at the beginning of deployment and lacks a coordination mechanism for dynamic expansion of business scale or elastic adjustment of hardware resources, this fixed design leads to a serious disconnect between storage strategies and real-time business needs and hardware load status. On the one hand, the mandatory implementation of high compression ratio strategies or high-frequency fragment merging during business peaks leads to non-linear growth in request response delays; while business slack periods result in low storage resource utilization and rising redundancy costs. On the other hand, static storage models cannot adapt to the dynamic expansion of hardware resources or changes in load. For example, after adding a new server node, the original compression ratio, overflow memory size and other parameters do not match the upgraded hardware performance, resulting in low resource utilization and limited system stability, which restricts the robustness and economy of distributed storage systems in ultra-large-scale scenarios. Summary of the Invention

[0004] In order to achieve dynamic and fine-grained load perception and data layout, the present invention provides a method and system for load perception and dynamic data layout based on deep learning. The technical solutions adopted are as follows:

[0005] The technical solution of the first aspect of the present invention provides a load perception and data dynamic layout method based on deep learning, the method comprising:

[0006] Collect hardware load indicators of each node in the distributed cluster in real time and pre-process the hardware load indicators;

[0007] Based on the pre-processed hardware load indicators, the comprehensive load score of each node is calculated using a weighted fuzzy logic algorithm;

[0008] Divide the load intervals according to the comprehensive load score and match them with predefined strategy combinations;

[0009] Based on the current business traffic peak and valley status of the target data block and the matching predefined strategy combination, the compression algorithm, compression ratio and fragment merging parameters are dynamically adjusted, where the business traffic peak and valley status is determined by the request processing rate and response time.

[0010] Furthermore, the hardware load indicators of each node in the distributed cluster are collected in real time and pre-processed, including:

[0011] The hardware load indicators of each node in the distributed cluster are collected in real time and normalized and preprocessed. The hardware load indicators include CPU utilization, CPU load, memory utilization, swap space utilization, disk utilization, I / O wait time, disk throughput, network bandwidth utilization, network latency, packet loss rate, number of active processes, response time, request processing rate and temperature.

[0012] Furthermore, the expression for calculating the comprehensive load score of each node is:

[0013]

[0014] Where S represents the comprehensive load score; w k represents the weight of the k-th hardware load indicator, represents the normalized value of the kth hardware load indicator; n represents the total number of hardware load indicators; α represents the temperature influence coefficient; T represents the current temperature; and T0 represents the temperature reference threshold.

[0015] Furthermore, the load range is divided according to the comprehensive load score of each node, including:

[0016] When S ≥ 80, it is classified as a high-load interval, indicating that the node is in a high-pressure state;

[0017] When 50≤S<80, it is classified as the medium load interval, indicating that the node is operating normally;

[0018] When S<50, it is classified as a low-load interval, indicating that the node resources are idle.

[0019] Furthermore, predefined strategy combinations are matched according to the load range, including:

[0020] The high-load interval is configured with the LZ4 compression algorithm, a compression ratio of 0.3, and fragment merging is disabled;

[0021] The medium load range is configured with the Zstandard compression algorithm, a compression ratio of 0.6, and a fragment merging period of 6 hours;

[0022] The low-load interval is configured with the Brotli compression algorithm, a compression ratio of 0.85, and a fragment merging period of 1 hour.

[0023] Furthermore, based on the current peak and valley status of the target data block's business traffic and the matching predefined strategy combination, the compression algorithm, compression ratio, and fragment merging parameters are dynamically adjusted, including:

[0024] When the disk usage is greater than a first preset value, the compression ratio is increased and the LZMA compression algorithm is switched;

[0025] When the memory usage is greater than a second preset value, the compression ratio is reduced by at least 0.1;

[0026] When the node temperature is greater than a third preset value, the fragment merging frequency is limited to 50% of the original value.

[0027] Furthermore, based on the current peak and valley status of the target data block's business traffic and the matching predefined strategy combination, the compression algorithm, compression ratio, and fragment merging parameters are dynamically adjusted, including:

[0028] A first elastic protection mechanism is configured to delay the fragment merging operation when the I / O waiting time is greater than a fourth preset value, and the delay duration is negatively correlated with the comprehensive load score;

[0029] The second elastic protection mechanism is configured to enable the LZMA compression algorithm to process the archived data when the comprehensive load score is less than a fifth preset value.

[0030] Furthermore, the method further comprises:

[0031] Feedback learning of the policy execution effect is periodically performed based on the Q-Learning algorithm to update the weights of each hardware load indicator in the weighted fuzzy logic algorithm. The reward function of the feedback learning is calculated based on compression time, space savings, CPU usage changes, and request latency changes.

[0032] Furthermore, the updated expression of the Q-Learning algorithm is:

[0033]

[0034] Where Q new (s,a) represents the expected cumulative reward value of taking action a under the current state s; s represents the current state, which indicates the hardware load score and business traffic status of the node; a represents the current action, which indicates the selected compression algorithm and parameters; η represents the learning rate; γ represents the discount factor; R(s,a) represents the reward function after the strategy is executed; Q(s ′ ,a ′ ) represents the next state s ′ Take the next action a ′ The expected cumulative reward value of s ′ Indicates the next state, that is, the state after executing the current action a; a′ Indicates the next action.

[0035] The technical solution of the second aspect of the present invention provides a load perception and data dynamic layout system based on deep learning, which adopts the load perception and data dynamic layout method based on deep learning described in the technical solution of the first aspect of the present invention. The system includes:

[0036] A data collection module is configured to collect hardware load indicators of each node in the distributed cluster in real time and pre-process the hardware load indicators;

[0037] A load score calculation module is configured to calculate the comprehensive load score of each node based on the preprocessed hardware load indicators using a weighted fuzzy logic algorithm;

[0038] a policy matching module configured to divide the load intervals according to the comprehensive load score and match the predefined policy combinations;

[0039] The dynamic execution module is configured to dynamically adjust the compression algorithm, compression ratio and fragment merging parameters based on the current business traffic peak and valley status of the target data block and the matching predefined strategy combination, wherein the business traffic peak and valley status is determined by the request processing rate and response time.

[0040] The reinforcement learning module is configured to periodically perform feedback learning on the policy execution effect based on the Q-Learning algorithm, and update the weight of each hardware load indicator in the weighted fuzzy logic algorithm.

[0041] The present invention has the following beneficial effects:

[0042] The deep learning-based load perception and data dynamic layout method provided by the present invention achieves fine-grained optimization of distributed storage systems by collecting multi-dimensional hardware load indicators in real time and quantifying the comprehensive load score, combined with dynamic load interval division and adaptive policy adjustment driven by business traffic. In actual use, it can prioritize response speed during business peak periods, reduce latency through low compression ratios and flexible merging strategies; improve compression efficiency to reduce storage redundancy during off-peak periods; and dynamically adapt strategies based on hardware load status to avoid resource bottlenecks and balance computing power distribution. Ultimately, it improves the system's adaptability to complex business scenarios, hardware resource utilization, and long-term operational stability, solving the rigidity and inefficiency of traditional static models in dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A flowchart of a method for load perception and dynamic data layout based on deep learning provided by one embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the structure of a load perception and data dynamic layout system based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] To further illustrate the technical means and effects employed by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effects of a method and system for load sensing and dynamic data layout based on deep learning proposed by the present invention. In the following description, different references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0047] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0048] The following describes in detail a specific solution of a load perception and data dynamic layout method and system based on deep learning provided by the present invention in conjunction with the accompanying drawings.

[0049] See also Figure 1 , which shows a method flow chart of a method for load perception and dynamic data layout based on deep learning provided by an embodiment of the present invention, the method comprising:

[0050] Step S100: collecting hardware load indicators of each node in the distributed cluster in real time and preprocessing the hardware load indicators. Specifically, collecting hardware load indicators of each node in the distributed cluster in real time and preprocessing the hardware load indicators to normalize them. Specifically, a lightweight monitoring agent is deployed on each node in the distributed cluster to obtain hardware load indicators in real time through the operating system interface and hardware sensors, with a collection frequency of once per second. The hardware load indicators include computing resources: CPU utilization (%), CPU load (1 / 5 / 15 minute average), memory utilization (%), and swap space utilization (%). Storage resources: disk utilization (%), I / O wait time (ms), disk throughput (MB / s); Network resources: network bandwidth utilization (%), network latency (ms), packet loss rate (%); Business status: number of active processes, request processing rate (req / s), response time (ms); Environmental parameters: node temperature (°C); Linear normalization is performed on the above hardware load indicators, and the original values of each indicator are mapped to the range of 0 and 1. For indicators without fixed upper limits such as disk throughput and network latency, this embodiment dynamically updates the indicator upper limit based on a sliding window to avoid normalization distortion caused by long-tail distribution. Pre-processed data storage includes: Cache mechanism: Temporarily store pre-processed data in an in-memory database or time series database for real-time call by the subsequent load score calculation module; Data backup: Periodically persist snapshot data to a distributed file system for offline analysis and model training.

[0051] This embodiment collects multi-dimensional hardware indicators in real time and performs normalization preprocessing to provide highly consistent, low-noise input data for subsequent load score calculations. This solves the evaluation bias problem caused by indicator dimensional differences and transient interference in traditional solutions. At the same time, through efficient transmission and caching mechanisms, it ensures that the system can still maintain millisecond-level response capabilities in ultra-large-scale clusters, laying a reliable data foundation for dynamic policy adjustments.

[0052] Step S200: Based on the pre-processed hardware load indicators, the comprehensive load score of each node is calculated by a weighted fuzzy logic algorithm. Specifically, this embodiment can use the hierarchical analysis method or the entropy weight method to assign initial weights to each hardware load indicator based on the business scenario type and historical load data. For example: high I / O business: disk throughput (0.25), I / O waiting time (0.2), CPU utilization (0.15), network delay (0.1), and the remaining weights are evenly distributed among the other indicators; computing-intensive business: CPU load (0.3), memory utilization (0.25), temperature (0.1), disk utilization (0.15), and the remaining indicators are evenly distributed.

[0053] Dynamic weight adjustment mechanism: The weight distribution is periodically optimized through Q-Learning feedback learning. The reward function is based on the compression time, resource savings, and latency changes after the policy is executed. The expression for calculating the comprehensive load score of each node is:

[0054]

[0055] Where S represents the comprehensive load score; w k represents the weight of the k-th hardware load indicator, represents the normalized value of the kth hardware load indicator; n represents the total number of hardware load indicators; α represents the temperature impact coefficient, α = 0.1, which means that for every 1°C above the temperature benchmark threshold, the comprehensive load score increases by 10%, to quantify the negative impact of high temperature on system stability; T represents the current temperature; T0 represents the temperature benchmark threshold, T0 = 70°C is the upper limit of the hardware safe operating temperature, which can be determined by referring to the server manufacturer's specifications and historical failure data; it should be noted that if the node temperature exceeds the safety threshold, S = 100 is directly marked and the load reduction strategy is triggered, such as shutting down non-critical processes; α and T0 can also be dynamically adjusted based on the deviation between the historical score and the actual load status, such as if the score is inflated but not actually overloaded. Specifically, the scoring error can be minimized by the gradient descent method.

[0056] This embodiment quantifies the comprehensive load status of nodes through a weighted fuzzy logic algorithm, integrating multi-dimensional heterogeneous hardware indicators into a single score, and solving the complexity and subjectivity of multi-dimensional load assessment in traditional solutions. Combined with a dynamic temperature correction mechanism, it accurately reflects the potential risks of high-temperature environments to hardware stability, providing a standardized and explainable load quantification basis for subsequent policy matching, thereby supporting fine-grained, scenario-adaptive dynamic adjustment of storage policies, and significantly improving the scientific nature of resource allocation and system robustness.

[0057] Step S300: Divide the load intervals according to the comprehensive load score and match them with predefined strategy combinations;

[0058] Step S300 specifically includes:

[0059] Step S310: When S ≥ 80, it is classified as a high-load interval, indicating that the node is under high pressure. Prioritize ensuring service response speed to avoid performance degradation due to resource contention.

[0060] When 50≤S<80, it is classified as a medium load range, indicating that the node is operating normally. It is necessary to balance performance and storage efficiency to adapt to mixed read and write scenarios.

[0061] When S<50, it is classified as a low-load interval, indicating that node resources are idle; this can maximize storage compression efficiency and optimize long-term storage costs.

[0062] Step S320: The high-load interval is configured with the LZ4 compression algorithm, a compression ratio of 0.3, and fragment merging is disabled; the medium-load interval is configured with the Zstandard compression algorithm, a compression ratio of 0.6, and a fragment merging period of 6 hours; the low-load interval is configured with the Brotli compression algorithm, a compression ratio of 0.85, and a fragment merging period of 1 hour; among them, the LZ4 compression algorithm is suitable for high-load scenarios, sacrificing compression rate in exchange for extremely fast compression / decompression, thereby reducing CPU usage; the Zstandard compression algorithm has a balanced compression ratio and speed, and is adapted to normal business loads; the Brotli compression algorithm has a high compression ratio but a slower speed, and is used in low-load periods or for archiving data; when the node load score changes across intervals, for example, from S=75 to S=45, a policy switch is triggered: terminate the current compression task, clear the pending queue; load new policy parameters, such as switching to the Brotli algorithm; and gradually adjust the compression ratio, such as 0.6→0.85, to avoid instantaneous resource impact. If the policy execution fails, such as the Brotli algorithm cannot be started due to insufficient memory, it will automatically fall back to the previous interval policy, such as the Zstandard compression algorithm, and trigger an alarm notification.

[0063] This embodiment achieves scenario-adaptive dynamic configuration of storage strategies by precisely matching quantitative load interval divisions with predefined policy combinations: in high-load intervals, low compression ratios and disabled fragment merging are used to ensure real-time business response, in medium-load intervals, performance and efficiency are balanced, and in low-load intervals, storage density is maximized. Combined with dynamic threshold calibration and anomaly fallback mechanisms, this ensures smooth and reliable policy switching, significantly improving the distributed storage system's adaptability to complex load scenarios while reducing manual parameter adjustment costs, providing support for resource utilization optimization and long-term operation and maintenance economy.

[0064] Step S400: Based on the current business traffic peak and valley state of the target data block and the matching predefined strategy combination, the compression algorithm, compression ratio and fragment merging parameters are dynamically adjusted, wherein the business traffic peak and valley state is determined by the request processing rate and response time; specifically, when the request processing rate exceeds the threshold and the response time is lower than the critical value, it is determined to be a business peak period, and a low compression ratio is preferentially used, such as the LZ4 algorithm, a compression ratio of 0.3 and disabling fragment merging, to minimize I / O blocking and computing overhead. When the request processing rate is lower than the threshold and the response time is higher than the critical value, it is determined to be a business off-peak period, and a high compression ratio strategy can be switched to, such as the Brotli compression algorithm, a compression ratio of 0.85 and enabling high-frequency fragment merging to optimize storage density.

[0065] Step S400 specifically includes:

[0066] Step S410: When the disk usage is greater than a first preset value, the compression ratio is increased and the LZMA compression algorithm is switched. Specifically, when the disk usage is greater than 90%, indicating that storage resources are tight, the compression ratio is increased to 0.95, and the LZMA algorithm is forcibly switched. At the same time, non-critical data writing is suspended, and expired data is cleared first.

[0067] When the memory usage is greater than a second preset value, the compression ratio is reduced by at least 0.1. Specifically, when the memory usage is greater than 95%, to prevent memory overflow, the compression ratio is dynamically reduced by at least 0.1, for example, from 0.6 to 0.5, to reduce memory usage. If the limit is still exceeded, the compression operation is temporarily disabled and the original data is directly stored.

[0068] When the node temperature is greater than a third preset value, the fragment merging frequency is limited to 50% of the original value; specifically, when the node temperature is >75°C, in order to avoid hardware overheating failure, the fragment merging frequency is limited to 50% of the original value, such as extending it from every 1 hour to every 2 hours, and reducing the priority of CPU-intensive tasks.

[0069] Step S420: A first elastic protection mechanism is configured to delay the fragment merging operation when the I / O wait time is greater than a fourth preset value, and the delay duration is negatively correlated with the comprehensive load score. Specifically, when the I / O wait time is greater than 50ms, indicating that the storage device is under a high load state, the fragment merging operation is delayed, and the delay duration is negatively correlated with the comprehensive load score, such as 12.5ms when S=80 and 16.7ms when S=60, to ensure that the merging task avoids the I / O peak window.

[0070] The second elastic protection mechanism is configured to enable the LZMA compression algorithm to process archived data when the comprehensive load score is less than a fifth preset value. Specifically, when the comprehensive load score is less than 30, indicating that node resources are extremely idle, the LZMA algorithm is enabled to process historical archived data, combined with low-frequency fragment merging to maximize long-term storage efficiency.

[0071] The compression ratio dynamic algorithm also includes:

[0072] When disk space is limited: the compression ratio is increased by 0.15, but not more than 0.95, to avoid CPU overload due to excessive compression;

[0073] When there is sufficient memory: the compression ratio is allowed to increase by 0.1, and the free memory is used to improve the compression efficiency;

[0074] Temperature control: When the temperature is >75°C, the fragment merging frequency is doubled to reduce hardware heating;

[0075] Load score driven: In the high load range, S≥80, fragment merging is disabled, and in the medium and low load ranges, it is executed according to the preset period.

[0076] This embodiment dynamically senses business traffic peaks and valleys and hardware resource status, adjusting compression algorithms, compression ratios, and fragment merging strategies in real time. This prioritizes low latency and high throughput during peak business periods, while maximizing storage efficiency during off-peak periods. It also incorporates elastic protection mechanisms to mitigate the risks of disk overflow, memory overload, and hardware overheating, enabling refined management of resource utilization. This embodiment breaks the rigid limitations of traditional static strategies and, through multi-dimensional adaptive adjustments, significantly improves the stability, responsiveness, and cost-effectiveness of distributed storage systems in complex and dynamic environments, while reducing manual operation and maintenance costs.

[0077] Preferably, the load perception and data dynamic layout method based on deep learning also includes:

[0078] Step S500: Periodically perform feedback learning on the policy execution effect based on the Q-Learning algorithm, and update the weights of each hardware load indicator in the weighted fuzzy logic algorithm. The reward function of the feedback learning is calculated based on compression time, space savings, CPU usage changes, and request latency changes. Specifically, the state space is defined as the combined composition of the node's comprehensive load score and the service traffic status (peak / valley value determination results). The action space is defined as a dynamically adjusted combination of storage policy parameters, including compression algorithm, compression ratio, and fragment merging frequency. The expression updated by the Q-Learning algorithm is:

[0079]

[0080] Where Q new (s,a) represents the expected cumulative reward value of taking action a under the current state s; s represents the current state, which indicates the hardware load score and business traffic status of the node; a represents the current action, which indicates the selected compression algorithm and parameters; η represents the learning rate, which ranges from 0.01 to 0.1; γ represents the discount factor, γ = 0.9; R(s,a) represents the reward function after the strategy is executed; Q(s ′ ,a ′ ) represents the next state s ′ Take the next action a ′ The expected cumulative reward value of s ′ Indicates the next state, that is, the state after executing the current action a; a ′ Indicates the next action; according to the updated Q value distribution, the index weight w of the weighted fuzzy logic algorithm in the reverse optimization step S200 k , the gradient descent method can be used to minimize the Q value prediction error;

[0081] Step S500 also includes: an experience replay mechanism, which randomly extracts historical data batches for training to break data correlation and prevent overfitting; an exploration-exploitation balance: in the exploration phase, actions are randomly selected with a probability of 10% to discover potentially better strategies; in the utilization phase, the action with the highest current Q value is selected with a probability of 90% to ensure strategy stability; it should be noted that if the new strategy causes key indicators such as latency and CPU usage to exceed safety thresholds, a rollback mechanism is triggered to restore to the previous stable version.

[0082] This embodiment uses the Q-Learning algorithm to build a closed-loop feedback learning system, dynamically mapping policy execution effects such as compression efficiency and delay changes to the weight optimization process, realizing a parameter-adaptive load assessment model; using discount factors to guide the model to balance immediate performance and resource utilization, avoiding local optimal traps; periodic training and weight adjustment enable the system to adapt to business model changes and hardware architecture upgrades; automated feedback learning reduces the need for manual parameter adjustment, reduces operation and maintenance complexity, and improves the scientific nature and consistency of policy decisions; experience replay and exception interception mechanisms effectively suppress model oscillations, ensuring the long-term stable operation of large-scale distributed clusters.

[0083] In summary, the load perception and data dynamic layout method based on deep learning provided by the present invention constructs a fine-grained load quantification system through multi-dimensional real-time data collection and weighted fuzzy logic scoring, and combines dynamic interval division with predefined strategy matching to achieve scenario-adaptive adjustment of storage parameters; further, through elastic optimization driven by business traffic and resource status, it prioritizes response speed in high-load scenarios and maximizes storage efficiency in low-load periods, while introducing disk, memory and temperature protection mechanisms to avoid resource overload risks; finally, through Q-Learning feedback learning, it periodically optimizes indicator weights and policy parameters, enabling the system to adapt to business model changes and hardware upgrades. The overall solution solves the rigidity problem of traditional static models, significantly improves the business adaptability, resource utilization and long-term stability of distributed storage systems in dynamic environments, and greatly reduces manual operation and maintenance costs through automated closed-loop optimization, providing technical support for the efficient and economical operation of ultra-large-scale clusters.

[0084] See also Figure 2 , which shows a schematic diagram of the structure of a load perception and data dynamic layout system based on deep learning provided by an embodiment of the present invention, the system includes:

[0085] A data collection module is configured to collect hardware load indicators of each node in the distributed cluster in real time and pre-process the hardware load indicators;

[0086] A load score calculation module is configured to calculate the comprehensive load score of each node based on the preprocessed hardware load indicators using a weighted fuzzy logic algorithm;

[0087] a policy matching module configured to divide the load intervals according to the comprehensive load score and match the predefined policy combinations;

[0088] The dynamic execution module is configured to dynamically adjust the compression algorithm, compression ratio and fragment merging parameters based on the current business traffic peak and valley status of the target data block and the matching predefined strategy combination, wherein the business traffic peak and valley status is determined by the request processing rate and response time.

[0089] The reinforcement learning module is configured to periodically perform feedback learning on the policy execution effect based on the Q-Learning algorithm, and update the weight of each hardware load indicator in the weighted fuzzy logic algorithm.

[0090] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0091] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A load perception and data dynamic layout method based on deep learning, characterized by: The method comprises: Collect hardware load indicators of each node in the distributed cluster in real time and pre-process the hardware load indicators; Based on the pre-processed hardware load indicators, the comprehensive load score of each node is calculated using a weighted fuzzy logic algorithm; Divide the load intervals according to the comprehensive load score and match them with predefined strategy combinations; Based on the current business traffic peak and valley status of the target data block and the matching predefined strategy combination, the compression algorithm, compression ratio and fragment merging parameters are dynamically adjusted, where the business traffic peak and valley status is determined by the request processing rate and response time.

2. The load perception and data dynamic layout method based on deep learning according to claim 1 is characterized in that: Real-time collection of hardware load indicators of each node in the distributed cluster and pre-processing of the hardware load indicators, including: The hardware load indicators of each node in the distributed cluster are collected in real time and normalized and preprocessed. The hardware load indicators include CPU utilization, CPU load, memory utilization, swap space utilization, disk utilization, I / O wait time, disk throughput, network bandwidth utilization, network latency, packet loss rate, number of active processes, response time, request processing rate and temperature.

3. The load perception and data dynamic layout method based on deep learning according to claim 1, characterized in that: The expression for calculating the comprehensive load score of each node is: Where S represents the comprehensive load score; w k represents the weight of the k-th hardware load indicator, represents the normalized value of the kth hardware load indicator; n represents the total number of hardware load indicators; α represents the temperature influence coefficient; T represents the current temperature; and T0 represents the temperature reference threshold.

4. The method for load perception and dynamic data layout based on deep learning according to claim 3, characterized in that: The load range is divided according to the comprehensive load score of each node, including: When S ≥ 80, it is classified as a high-load interval, indicating that the node is in a high-pressure state; When 50≤S<80, it is classified as the medium load interval, indicating that the node is operating normally; When S<50, it is classified as a low-load interval, indicating that the node resources are idle.

5. The method for load perception and dynamic data layout based on deep learning according to claim 4, characterized in that: Match predefined strategy combinations based on load ranges, including: The high-load interval is configured with the LZ4 compression algorithm, a compression ratio of 0.3, and fragment merging is disabled; The medium load range is configured with the Zstandard compression algorithm, a compression ratio of 0.6, and a fragment merging period of 6 hours; The low-load interval is configured with the Brotli compression algorithm, a compression ratio of 0.85, and a fragment merging period of 1 hour.

6. The method for load perception and dynamic data layout based on deep learning according to claim 1 is characterized in that: Based on the current business traffic peak and valley status of the target data block and the matching predefined strategy combination, the compression algorithm, compression ratio and fragment merging parameters are dynamically adjusted, including: When the disk usage is greater than a first preset value, the compression ratio is increased and the LZMA compression algorithm is switched; When the memory usage is greater than a second preset value, the compression ratio is reduced by at least 0.1; When the node temperature is greater than a third preset value, the fragment merging frequency is limited to 50% of the original value.

7. The method for load perception and dynamic data layout based on deep learning according to claim 6 is characterized in that: Based on the current peak and valley status of the target data block's business traffic and the matching predefined strategy combination, the compression algorithm, compression ratio, and fragment merging parameters are dynamically adjusted. This also includes: A first elastic protection mechanism is configured to delay the fragment merging operation when the I / O waiting time is greater than a fourth preset value, and the delay duration is negatively correlated with the comprehensive load score; The second elastic protection mechanism is configured to enable the LZMA compression algorithm to process the archived data when the comprehensive load score is less than a fifth preset value.

8. The load perception and data dynamic layout method based on deep learning according to any one of claims 1 to 7, characterized in that: The method further comprises: Feedback learning of the policy execution effect is periodically performed based on the Q-Learning algorithm to update the weights of each hardware load indicator in the weighted fuzzy logic algorithm. The reward function of the feedback learning is calculated based on compression time, space savings, CPU usage changes, and request latency changes.

9. The method for load perception and dynamic data layout based on deep learning according to claim 8, characterized in that: The expression for Q-Learning algorithm update is: Where Q new (s,a) represents the expected cumulative reward value of taking action a in the current state s; s represents the current state, which indicates the hardware load score and business traffic status of the node; a represents the current action, which indicates the selected compression algorithm and parameters; η represents the learning rate; γ represents the discount factor; R(s,a) represents the reward function after the strategy is executed; Q(s′,a′) represents the expected cumulative reward value of taking the next action a′ in the next state s′; s′ represents the next state, that is, the state after executing the current action a; a′ represents the next action.

10. The load perception and data dynamic layout system based on deep learning is characterized by: The method for load perception and dynamic data layout based on deep learning according to any one of claims 1 to 9 is adopted, wherein the system comprises: A data collection module is configured to collect hardware load indicators of each node in the distributed cluster in real time and pre-process the hardware load indicators; A load score calculation module is configured to calculate the comprehensive load score of each node based on the preprocessed hardware load indicators using a weighted fuzzy logic algorithm; a policy matching module configured to divide the load intervals according to the comprehensive load score and match the predefined policy combinations; The dynamic execution module is configured to dynamically adjust the compression algorithm, compression ratio and fragment merging parameters based on the current business traffic peak and valley status of the target data block and the matching predefined strategy combination, wherein the business traffic peak and valley status is determined by the request processing rate and response time. The reinforcement learning module is configured to periodically perform feedback learning on the policy execution effect based on the Q-Learning algorithm, and update the weight of each hardware load indicator in the weighted fuzzy logic algorithm.

Images