Cloud computing platform data storage method based on big data analysis

Abstract

The invention discloses a cloud computing platform data storage method based on big data analysis, and relates to the technical field of cloud computing data storage. The technical problems that according to a traditional cloud computing storage system, resource configuration is homogenized, hot data access delay is too high, cost is wasted due to the fact that cold data occupies high-performance resources, a data popularity dynamic sensing and self-adaptive scheduling mechanism is lacked, and load distribution of storage nodes is unbalanced are solved. Compared with a traditional cloud computing unified storage mode, the method has the advantages that a whole-process technical system of storage resource hierarchical division, big data access frequency analysis, dynamic hierarchical storage, real-time migration scheduling and load balancing calibration is constructed; low-time-delay high-speed access of high-frequency hot spot data, performance and cost balance of medium-frequency temperature spot data and low-cost storage of low-frequency cold spot data are achieved, and storage resources are always accurately matched with data popularity through big data real-time analysis and a self-adaptive scheduling mechanism.

Description

Technical Field

[0001] This invention relates to the field of cloud computing data storage technology, specifically to a cloud computing platform data storage method based on big data analysis. Background Technology

[0002] In the context of the comprehensive advancement of digital transformation, the scale of data generated across various industries is experiencing explosive growth. Scenarios such as big data processing, artificial intelligence training, internet business services, and core enterprise systems all heavily rely on the storage and computing services provided by cloud computing platforms. With its advantages of elastic scalability, on-demand allocation, and centralized management, cloud computing platforms have become the core carrier for massive data storage, data circulation, and business support. As the underlying foundation of cloud computing platforms, the read / write performance, response latency, resource utilization, and storage costs of data storage systems directly determine the operational efficiency, service quality, and overall competitiveness of upper-layer businesses.

[0003] Current traditional cloud computing platforms generally suffer from simplistic design approaches and rigid resource scheduling in their data storage solutions. The vast majority of cloud storage systems employ a homogeneous storage architecture with uniform hardware configurations, redundancy strategies, and bandwidth allocation, failing to differentiate and match resources based on the access frequency, business importance, and read / write characteristics of different data. In actual operation, frequently accessed hot data, moderately accessed lukewarm data, and rarely accessed cold data are indiscriminately stored on storage nodes of the same performance level. This leads to a series of technical defects that are difficult to resolve through simple optimization: First, hot data, under high concurrency and high traffic scenarios, is prone to input / output blocking, bandwidth contention, and increased read / write queuing latency, directly resulting in slow business response, degraded user experience, and even stability issues such as service timeouts and request failures during peak traffic periods. Second, a large amount of long-term inaccessible cold data, which has no requirements for read / write performance, continuously occupies high-performance storage resources with high input / output operations per second, multiple replicas, and high bandwidth, causing severe hardware resource redundancy. First, equipment investment costs and data center energy consumption costs remain high, resulting in a severe mismatch between resource input and actual returns. Second, existing storage systems lack real-time access behavior analysis capabilities based on big data, making it impossible to continuously collect, quantify, and dynamically perceive data access frequency. Furthermore, they cannot perform automated and intelligent storage location scheduling based on changes in data popularity, leaving storage strategies in a static and fixed state, resulting in overall low storage efficiency. Third, storage node load cannot achieve global balance. High-level storage nodes operate under high load for extended periods due to the concentration of hot data, while low-level storage nodes remain idle with extremely low resource utilization. This severe imbalance in node load distribution reduces the overall scalability, reliability, and resilience of the cloud computing platform.

[0004] To address a series of problems in existing technologies, such as unreasonable resource allocation, high latency for hot data, high cost for cold data, lack of dynamic scheduling, and unbalanced load, this invention proposes a data storage method for cloud computing platforms based on big data analytics. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a data storage method for cloud computing platforms based on big data analytics, which solves the technical problems of homogeneous resource configuration, excessively high latency in accessing hot data, cost waste caused by cold data occupying high-performance resources, lack of dynamic data popularity perception and adaptive scheduling mechanism, and uneven load distribution of storage nodes in traditional cloud computing storage systems.

[0006] To achieve the above objectives, the present invention provides a data storage method for a cloud computing platform based on big data analytics, comprising the following steps:

[0007] Step 1: Storage Resource Classification: The cloud computing platform storage nodes are classified into high-level (S1), medium-level (S2), and low-level (S3) based on input / output performance, number of replicas, bandwidth, and redundancy strategy. S1 employs a strong redundancy strategy with 3 or more replicas, S2 employs a balanced redundancy strategy with 2 replicas, and S3 employs a redundancy strategy with 1 replica or erasure coding. S1 is used for high-frequency hot data, S2 for medium-frequency warm data, and S3 for low-frequency cold data. The storage resource levels satisfy the following order: number of high-level S1 nodes > number of medium-level S2 nodes > number of low-level S3 nodes. Higher levels result in higher input / output operations per second per node, more replicas, greater access bandwidth, and lower response latency.

[0008] Step 2: Data Access Frequency Collection and Calculation: Access logs of all data objects are collected in real-time using the big data collection module. Access frequency is counted according to a sliding time window, and data popularity is calculated using the average access frequency formula. The average access frequency formula is: F i = Among them, F i The average access frequency of the i-th data object; The number of times the i-th data is accessed in the t-th time unit; T is the sliding window duration, which is a preset constant.

[0009] Step 3: Data popularity classification determination: Set a high-frequency threshold F H Low-frequency threshold F L The data is divided into hot data, moderate data, and cold data, and each data object is bound with a real-time updatable popularity tag; the data popularity classification judgment rule is as follows:

[0010] Hot data: F i ≥F H ;

[0011] Temperature point data: F L <F i <F H ;

[0012] Cold spot data, F i ≤F L ;

[0013] Among them, F H For high-frequency threshold, F L This is the low-frequency threshold.

[0014] Step 4: The central controller of the cloud computing platform automatically stores hot data into S1, lukewarm data into S2, and cold data into S3 according to the correspondence between the data heat labels and the storage resource levels.

[0015] Step 5: Real-time monitoring and dynamic migration of access frequency: The average access frequency is refreshed at a fixed period, and the absolute value of the change between the current popularity value and the historical popularity value is calculated. When the popularity change reaches the preset frequency change threshold, data migration across storage levels is automatically triggered to adapt the data to the latest popularity level.

[0016] The specific method for cross-storage level data migration is as follows: △F i =∣F i New-F i Old∣when△Fᵢ≥△F 阈值 Time-triggered migration, △F 阈值 For the frequency change threshold, ΔF i F represents the absolute value of the change in temperature. i New represents the latest popularity value for the current period, F i The previous period's historical popularity value;

[0017] Originally cold or lukewarm data has been upgraded to hot data and migrated to S1;

[0018] Data that was originally hot or lukewarm was downgraded to cold data and migrated to S3;

[0019] Data that was originally hotspot data has been downgraded to lukewarm data and migrated to S2;

[0020] The original cold point data was upgraded to warm point data and migrated to S2.

[0021] The dynamic migration process follows the following coordination priority rules:

[0022] When data is upgraded from cold or warm data to hot data, the operation of migrating to high-level S1 storage nodes is performed first to ensure the high-speed access needs of core businesses.

[0023] When high-level S1 and medium-level S2 storage nodes are short of resources or overloaded, cold data within the nodes will be migrated out of the S1 and S2 nodes first to release high-performance storage resources.

[0024] Sort the data objects by access frequency from low to high, and migrate the data objects with the lowest access frequency first.

[0025] Step 6: Storage Node Load Balancing Calibration: Calculate the load rate of nodes in each level. When the load exceeds the upper limit threshold, migrate the data with the lowest access frequency in that level to the next level.

[0026] The node load rate L j It is calculated by weighting CPU utilization, disk I / O utilization, network bandwidth utilization, and memory usage. When L j >L max Start load calibration at time, L max The preset node load safety limit threshold for the system.

[0027] This invention provides a data storage method for a cloud computing platform based on big data analytics. Compared with existing technologies, it has the following advantages:

[0028] This invention constructs a complete technical system encompassing "hierarchical partitioning of storage resources, analysis of big data access frequency, dynamic hierarchical storage, real-time migration scheduling, and load balancing calibration." Compared to the traditional unified storage model of cloud computing, it achieves low-latency, high-speed access to high-frequency hot data, a performance-cost balance for mid-frequency warm data, and low-cost storage for low-frequency cold data. Furthermore, through real-time big data analysis and adaptive scheduling mechanisms, it ensures that storage resources are always precisely matched with data popularity. This effectively solves problems such as unreasonable resource allocation, slow response to hot data, high cost of cold data, and unbalanced node load in existing technologies, significantly improving the storage efficiency, resource utilization, and business service capabilities of cloud computing platforms. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the principle framework of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] The data storage method for cloud computing platforms based on big data analytics proposed in this invention allows for unified global scheduling by the platform's central controller during actual deployment and operation. The big data acquisition and analysis module is responsible for access log collection, access frequency statistics, popularity value calculation, popularity classification, migration condition judgment, and load data calculation. Multi-level differentiated storage nodes are responsible for the actual reading, writing, storage, and migration of data. The entire method can achieve fully automatic and intelligent operation 24 / 7 without manual intervention and without interrupting the normal service of upper-layer businesses.

[0032] Step 1: Storage Resource Tier Classification

[0033] During the initial deployment phase of the cloud computing platform, all available storage nodes within the platform are first uniformly assessed, their performance tested, and standardized into tiers. The tiering is based on several key indicators, including the number of input / output operations per second per node, the number of data replicas configured, maximum available access bandwidth, data redundancy protection strategies, hardware storage media type, unit storage cost, and average read / write latency. Storage nodes are uniformly classified into three tiers—High-Tier S1, Mid-Tier S2, and Low-Tier S3—according to performance and cost from highest to lowest. The specific definitions for each tier are as follows:

[0034] High-level S1 storage nodes: Adopt enterprise-grade high-performance SSD solid-state drives, configured with a strong data redundancy protection strategy of 3 replicas or more, with ultra-high input / output operations per second, ultra-large bandwidth, ultra-low access latency, and high concurrency carrying capacity. They are specially designed to store high-frequency access and hot data with extremely high response speed requirements, ensuring fast and stable response in high-concurrency scenarios of core business.

[0035] Mid-level S2 storage node: It adopts a hybrid storage medium of SSD and SAS, and is configured with a 2-replica balanced redundancy strategy. Its input / output performance, access bandwidth and hardware cost are all at a medium level. It is suitable for storing warm data with moderate access frequency and relatively balanced performance requirements, balancing storage performance and hardware cost.

[0036] Low-level S3 storage nodes: use large-capacity, low-cost SATA hard drives, configured with 1-replica or erasure coding redundancy strategies, have high storage density, extremely low unit storage cost, and relatively low input / output performance. They are suitable for storing low-frequency, cold data that is rarely accessed, is not sensitive to read / write latency, and needs to be stored for a long time.

[0037] To ensure reasonable resource allocation, efficient scheduling, and system stability, this invention sets strict constraints on storage resource levels:

[0038] The number of high-level S1 nodes > the number of medium-level S2 nodes > the number of low-level S3 nodes, prioritizing the supply of sufficient high-performance resources for hot data;

[0039] The higher the storage level, the more I / O operations per second per disk on the corresponding node, the more data replicas, the larger the allocated available bandwidth, and the lower the average access latency. Performance is strictly positively correlated with the level.

[0040] Different levels of storage nodes operate with logical or physical isolation to avoid interference and resource contention between data with different usage levels, thus ensuring the stability of storage services at each level.

[0041] After the classification is completed, the platform's central controller will write the classification information, performance parameters, load limit threshold, available capacity, and other information of each node into the system configuration library for subsequent scheduling, calculation, and migration processes.

[0042] Step 2: Data Access Frequency Acquisition and Calculation

[0043] The big data acquisition module collects all access log information of all data objects within the cloud computing platform in real time through bypass monitoring or standardized interface calls. The collected information includes unique data identifiers (IDs), access timestamps, access operation types, access user identifiers, source IP addresses, and data read / write sizes. After cleaning, deduplicating, and standardizing the logs, the acquisition module proceeds to the access frequency statistics and calculation stage.

[0044] This invention employs a sliding time window T mechanism to continuously statistically analyze access behavior. The window duration T is a configurable preset constant that can be flexibly set to different durations such as 5 minutes, 10 minutes, 30 minutes, and 1 hour according to business scenarios, to adapt to the access fluctuation characteristics of different businesses such as e-commerce, finance, IoT, and enterprise office. After each sliding time window is completed, the access popularity of each data object is immediately calculated numerically using the average access frequency formula, as follows: F i = The parameters in the formula are defined as follows:

[0045] F i The average access frequency of the i-th data object is the quantified popularity value of that data.

[0046] The number of times the i-th data is accessed within the t-th time unit;

[0047] T represents the duration of the sliding window, which is a system-preset constant.

[0048] The access frequency collection and popularity calculation process is executed in real-time in a loop. A full data popularity update is completed after each sliding time window, and the data is synchronized to the platform's central controller in real time, providing accurate data for subsequent classification, storage, and migration. In the calculation process of the above formula, all parameters are stripped of physical dimensions, and only pure numerical calculations are performed.

[0049] Step 3: Determine the data popularity level

[0050] The big data analytics module pre-sets a high-frequency threshold F based on the platform's business importance, performance requirements, and resource allocation strategies. H With low frequency threshold F L The threshold can be dynamically adjusted based on business load, peak access frequency, and storage size. The average access frequency F calculated in step 2 is used as an example. i Based on this, all data objects are strictly and standardized in terms of popularity classification. The specific judgment rules are as follows: hot data, moderate data, and cold data. After the classification is completed, the big data analysis module binds a unique popularity tag to each data object. The tag content includes data ID, current popularity level, the time of the most recent popularity calculation, and current popularity value F. i Tag information is synchronized in real time to the platform's central controller and storage management module, serving as the direct basis for storage allocation and migration scheduling.

[0051] Step 4: Initial hierarchical storage mapping

[0052] The platform's central controller automatically executes initial hierarchical storage mapping rules based on the popularity tags carried by data objects, allocating data of different popularity levels to the most suitable storage level nodes to achieve optimal matching between data and storage resources. The specific mapping relationship is as follows:

[0053] Hot data is automatically allocated and stored in a high-level S1 high-performance storage node cluster;

[0054] Temperature data is automatically allocated and stored in a mid-tier S2 balanced storage node cluster;

[0055] Cold spot data - automatically allocated and stored to a low-tier S3 low-cost storage node cluster.

[0056] Once the initial storage allocation is complete, the data immediately enters a normal read / write service state, and upper-layer businesses can access the data normally. The big data analytics module continuously monitors changes in access frequency in the background without interruption or impact on business operations.

[0057] Step 5: Real-time monitoring and dynamic migration of access frequency

[0058] To adapt to the dynamic changes in data popularity caused by business fluctuations, peak access times, and changes in user behavior, this invention sets a fixed monitoring cycle, cyclically refreshes the average access frequency Fᵢ of all data objects, and performs cross-level migration judgment. The judgment formula is as follows: △F i =∣F i New-F i Parameter definition in the old | formula:

[0059] △Fi The absolute value of the change in data popularity represents the magnitude of the increase or decrease in popularity.

[0060] F i New: The latest popularity value calculated for the current monitoring period;

[0061] F i Old: Historical heat value calculated from the previous monitoring period.

[0062] The system pre-sets a frequency change threshold ΔF 阈值 When △F i ≥△F 阈值 If the data's popularity has changed significantly and it is no longer suitable to be stored on the current level node, a cross-level migration process will be automatically triggered. The specific migration execution rules are as follows:

[0063] Originally cold or lukewarm data has been upgraded to hot data and migrated to S1;

[0064] Data that was originally hot or lukewarm was downgraded to cold data and migrated to S3;

[0065] Data that was originally hotspot data has been downgraded to lukewarm data and migrated to S2;

[0066] The original cold point data was upgraded to warm point data and migrated to S2.

[0067] To ensure migration efficiency, resource utilization, and business stability, the dynamic migration process strictly follows the following priority rules:

[0068] Hot data is prioritized for migration to the S1 node to ensure high-speed access for core businesses.

[0069] Cold data is migrated out of S1 and S2 nodes first to free up high-performance storage resources;

[0070] Within the same storage tier, data is sorted by access frequency from low to high, and the data with the lowest access frequency is migrated first.

[0071] The migration process is executed asynchronously in the background, ensuring uninterrupted and smooth data read / write services, and leaving the upper-layer business processes unaware of the changes.

[0072] Step 6: Storage Node Load Balancing Calibration

[0073] To avoid performance degradation, service instability, or even downtime due to excessive load on a single storage node, the platform collects comprehensive operational status data for each storage node in real time, including CPU utilization, disk I / O utilization, network bandwidth utilization, memory usage, and remaining disk space. The overall node load rate L is then calculated using a weighted average method. j .

[0074] The system has a pre-set load limit threshold L max When the overall load rate L of a certain level of storage nodes j >L max If the node is found to be overloaded, the load balancing calibration process is immediately initiated: within the current level of storage nodes, the data objects with the lowest access frequency are selected and migrated down one storage level (S1→S2, S2→S3) until the overall load rate of the node drops to within the safe threshold range.

[0075] The load balancing calibration process and the dynamic migration process are executed synchronously and in parallel, cooperating with each other without conflict, to jointly ensure that the cloud computing platform storage system is in a stable, efficient and balanced operating state in the long term.

[0076] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0077] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A data storage method for a cloud computing platform based on big data analytics, characterized in that, Includes the following steps: Step 1: Storage Resource Classification: The cloud computing platform storage nodes are classified into high-level (S1), medium-level (S2), and low-level (S3) based on input / output performance, number of replicas, bandwidth, and redundancy strategy. S1 adopts a strong redundancy strategy with 3 or more replicas, S2 adopts a balanced redundancy strategy with 2 replicas, and S3 adopts a redundancy strategy with 1 replica or erasure coding. S1 is used for high-frequency hot data, S2 is used for medium-frequency warm data, and S3 is used for low-frequency cold data. Step 2: Data Access Frequency Collection and Calculation: The access logs of all data objects are collected in real time through the big data collection module. The number of accesses is counted according to the sliding time window, and the data popularity is calculated using the average access frequency formula. Step 3: Data popularity classification determination: Set a high-frequency threshold F H Low-frequency threshold F L The data is divided into hot data, lukewarm data, and cold data, and each data object is bound with a heat tag that can be updated in real time. Step 4: The central controller of the cloud computing platform automatically stores hot data into S1, lukewarm data into S2, and cold data into S3 according to the correspondence between the data heat labels and the storage resource levels. Step 5: Real-time monitoring and dynamic migration of access frequency: The average access frequency is refreshed at a fixed period, and the absolute value of the change between the current popularity value and the historical popularity value is calculated. When the popularity change reaches the preset frequency change threshold, data migration across storage levels is automatically triggered to adapt the data to the latest popularity level. Step Six: Storage Node Load Balancing Calibration: Calculate the load rate of nodes at each level, node load rate L j It is calculated by weighting CPU utilization, disk I / O utilization, network bandwidth utilization, and memory usage. When L j >L max Start load calibration at time, L max The preset node load safety limit threshold for the system; When the load exceeds the upper limit threshold, the data with the lowest access frequency within that level is migrated down one level.

2. The data storage method for a cloud computing platform based on big data analytics according to claim 1, characterized in that, The storage resource levels satisfy the following criteria: number of high-level S1 nodes > number of medium-level S2 nodes > number of low-level S3 nodes. The higher the level, the higher the number of input / output operations per second per node, the more replicas, the greater the access bandwidth, and the lower the response latency.

3. The data storage method for a cloud computing platform based on big data analytics according to claim 1, characterized in that, The formula for the average access frequency is: F i = Among them, F i The average access frequency of the i-th data object; The number of times the i-th data is accessed in the t-th time unit; T is the sliding window duration, which is a preset constant.

4. The data storage method for a cloud computing platform based on big data analytics according to claim 3, characterized in that, The data popularity grading determination rule is as follows: Hot data: F i ≥F H ; Temperature point data: F L <F i <F H ; Cold spot data, F i ≤F L ; Among them, F H For high-frequency threshold, F L This is the low-frequency threshold.

5. The data storage method for a cloud computing platform based on big data analytics according to claim 4, characterized in that, The specific method for cross-storage level data migration is as follows: △F i =∣F i New-F i Old∣when△Fᵢ≥△F 阈值 Time-triggered migration, △F 阈值 For the frequency change threshold, ΔF i F represents the absolute value of the change in temperature. i New represents the latest popularity value for the current period, F i The previous period's historical popularity value; Originally cold or lukewarm data has been upgraded to hot data and migrated to S1; Data that was originally hot or lukewarm was downgraded to cold data and migrated to S3; Data that was originally hotspot data has been downgraded to lukewarm data and migrated to S2; The original cold point data was upgraded to warm point data and migrated to S2.

6. The data storage method for a cloud computing platform based on big data analytics according to claim 5, characterized in that, The dynamic migration process follows the following coordination priority rules: When data is upgraded from cold or warm data to hot data, the operation of migrating to high-level S1 storage nodes is performed first to ensure the high-speed access needs of core businesses. When high-level S1 and medium-level S2 storage nodes are short of resources or overloaded, cold data within the nodes will be migrated out of the S1 and S2 nodes first to release high-performance storage resources. Sort the data objects by access frequency from low to high, and migrate the data objects with the lowest access frequency first.

Images