An optimization method and system for intelligent cloud data management

Abstract

This invention relates to an optimization method and system for intelligent cloud data management, belonging to the field of intelligent cloud data management technology. It includes the following steps: S1: Data acquisition and preprocessing; S2: Intelligent hierarchical storage optimization; S3: Metadata load balancing optimization; S4: Data access optimization; S5: Real-time monitoring and dynamic adjustment. This invention, through intelligent hierarchical storage, allocates data of varying frequency to storage nodes with corresponding costs, reducing waste of high-speed storage resources and lowering the cost of storing cold data. Compared to traditional storage methods, storage costs can be significantly reduced. It covers the entire process of data acquisition, storage, access, and monitoring, optimizing multiple stages to improve the overall efficiency of cloud data management. Through metadata load balancing, intelligent caching, and transmission optimization, it reduces data access latency, improves cache hit rate, and effectively enhances data access efficiency.

Description

Technical Field

[0001] This invention relates to an optimization method and system for intelligent cloud data management, belonging to the field of intelligent cloud data management technology. Background Technology

[0002] With the rapid development of cloud computing technology and the deepening of digital transformation, the amount of cloud data generated by enterprises and individuals is growing explosively, making the efficiency, cost, and security of cloud data management core requirements. Intelligent Cloud Data Management is a product of the integration of cloud computing, big data, and artificial intelligence (AI) technologies. Essentially, it is a "data brain" deployed in the cloud, aiming to solve the pain points of enterprises' massive amounts of data being "unable to be stored, managed, or utilized effectively." Its core feature is a shift from passive storage to proactive governance and intelligent services.

[0003] While existing intelligent cloud data management systems can generally meet normal usage needs, several pain points remain: First, data storage strategies are rigid. Current technologies often employ a single storage model, failing to tier storage based on data access frequency, update frequency, and importance. This results in high-frequency access (hot data) sharing high-speed storage resources with low-frequency access (cold data), leading to significant waste of high-speed storage resources and cold data occupying high-cost storage nodes for extended periods, substantially increasing the overall cost of cloud data storage. Second, metadata management and data access efficiency are low. In distributed cloud storage systems, metadata acts as an index, and its access efficiency directly determines overall data access performance. However, existing metadata management often uses static allocation, neglecting the locality of metadata access and dynamic load changes. This easily leads to metadata access hotspots, causing some storage nodes to be overloaded, increasing data access latency, and even causing access congestion. Consequently, during data access, high transmission bandwidth consumption and unreasonable caching strategies result in low transmission rates, impacting access efficiency. Therefore, this invention provides an optimized method and system for intelligent cloud data management. Summary of the Invention

[0004] The technical problems to be solved by this invention are: rigid data storage strategies, the use of a single storage mode, which increases storage costs; low efficiency in metadata management and data access.

[0005] To address the aforementioned technical problems, this invention provides an optimized method for intelligent cloud data management. The proposed technical solution includes the following steps:

[0006] S1: Data Acquisition and Preprocessing: Collect multi-source heterogeneous data from the cloud platform, perform deduplication, noise reduction, and format standardization on the data to generate structured data; collect data access characteristics in real time through PEBS (Performance Event Based Sampling) technology, and label the data popularity level with data attributes, classifying it into three categories: hot data, warm data, and cold data;

[0007] S2: Intelligent Tiered Storage Optimization: Based on data popularity level and access frequency, hot data is stored in high-speed cloud storage nodes, warm data is stored in regular cloud storage nodes, and cold data is migrated to low-cost archive storage nodes. A dynamic migration mechanism is established, and data popularity level is updated and storage location is adjusted every preset decision monitoring cycle.

[0008] S3: Metadata load balancing optimization: Build a machine learning-driven metadata management framework to collect metadata access load data in real time. Based on the locality characteristics and hierarchical structure of metadata, dynamically allocate metadata storage nodes to avoid access hotspots and reduce metadata operation latency.

[0009] S4: Data Access Optimization: Establish an intelligent caching mechanism to cache frequently accessed hot data and metadata to edge nodes, and adjust the caching strategy in conjunction with the eBPF (Extended Berkeley Packet Filter) mechanism to reduce the data transmission pressure in the cloud center; at the same time, optimize the data transmission protocol and use compression coding technology to reduce the transmission bandwidth consumption;

[0010] S5: Real-time monitoring and dynamic adjustment: Real-time monitoring of data storage status, access efficiency, and system load; setting threshold early warning mechanisms; and automatically adjusting storage node allocation, caching strategies, and data migration schemes when an anomaly is detected to ensure stable and efficient system operation.

[0011] Furthermore, in step S1, the data preprocessing also includes missing value imputation, which uses an interpolation method based on data distribution to fill in missing data; the data popularity level is comprehensively quantified and evaluated by access frequency, access duration, and data update frequency, and the weight of the quantification is dynamically adjusted according to user needs.

[0012] Furthermore, in step S2, the dynamic migration mechanism adopts an incremental migration method, which only migrates data whose popularity level has changed. During the migration process, breakpoint resume technology is used to avoid data loss, and the migration operation is performed during periods of low system load, so as not to affect normal data access.

[0013] Furthermore, in step S3, the machine learning-driven metadata management framework includes a data collection unit, a model training unit, and a decision-making unit. The model training unit uses the gradient boosting tree algorithm to train a load prediction model based on historical access data. The decision-making unit realizes metadata load balancing distribution based on the prediction results.

[0014] Furthermore, in step S4, the compression coding technology adopts an adaptive compression algorithm, which automatically selects the optimal compression method according to the data type. Text data adopts the LZ77 (Lempel-Ziv 77) algorithm, and binary data adopts the Huffman coding algorithm, with the compression ratio controlled between 1:3 and 1:5.

[0015] An optimization system for intelligent cloud data management, used to implement the aforementioned optimization method for intelligent cloud data management, includes:

[0016] Data acquisition and preprocessing module: used to collect multi-source heterogeneous data from the cloud platform, perform deduplication, noise reduction, format standardization, missing value imputation, label data popularity level, and output structured data;

[0017] Intelligent hierarchical storage module: connected to the data acquisition and preprocessing module, it allocates storage nodes based on the data popularity level, realizing dynamic migration and hierarchical storage of hot, warm and cold data, including high-speed storage nodes, regular storage nodes and archive storage nodes;

[0018] Metadata load balancing module: Connected to the intelligent tiered storage module, it is used to collect metadata access load data and realize the dynamic allocation of metadata storage nodes through machine learning models to optimize metadata access efficiency;

[0019] Data access optimization module: It connects to the intelligent hierarchical storage module and the metadata load balancing module respectively, builds an intelligent caching mechanism, optimizes the data transmission protocol, and adopts compression encoding technology to improve data access and transmission efficiency;

[0020] Real-time monitoring and control module: Connected to the data acquisition and preprocessing module, intelligent hierarchical storage module, metadata load balancing module, and data access optimization module, it is used to collect system operating parameters, data storage status, and access data in real time, set threshold warnings, automatically adjust system optimization strategies, and ensure stable system operation.

[0021] Furthermore, the high-speed storage node uses an SSD solid-state drive array, the conventional storage node uses a SAS hard disk storage array, and the archive storage node uses a combination of tape library and cloud archive storage to reduce storage costs.

[0022] Furthermore, the real-time monitoring and control module includes a monitoring unit, an early warning unit, and a control unit. The monitoring unit collects parameters such as storage utilization, data access latency, and system load in real time. The early warning unit issues an early warning signal when the parameters exceed a preset threshold. The control unit automatically adjusts storage allocation and optimization strategies.

[0023] Furthermore, the data acquisition and preprocessing module also includes a data encryption unit, which uses the AES-256 (Advanced Encryption Standard 256-bit) encryption algorithm to encrypt the acquired sensitive data, ensuring the security of data transmission and storage.

[0024] Furthermore, it also includes a user interaction module, which receives parameter setting instructions from the user, displays the system's operating status and data optimization results, and supports users to manually adjust optimization strategies.

[0025] The beneficial effects of this invention are:

[0026] By using intelligent tiered storage, data of varying popularity is allocated to storage nodes with corresponding costs, reducing the waste of high-speed storage resources and lowering the cost of cold data storage. Compared to traditional storage methods, storage costs can be significantly reduced. Covering the entire process of data collection, storage, access, and monitoring, multiple aspects are optimized to improve the overall efficiency of cloud data management. Through metadata load balancing, intelligent caching, and transmission optimization, data access latency is reduced, cache hit rate is improved, and data access efficiency can be effectively enhanced. Attached Figure Description

[0027] Figure 1 This is a flowchart of the overall process of the present invention.

[0028] Figure 2 This is the data acquisition and preprocessing sub-process of the present invention.

[0029] Figure 3 This is the data dynamic migration sub-process of the present invention.

[0030] Figure 4 This is a block diagram showing the overall system module connections of the present invention.

[0031] Figure 5 This is an internal block diagram of the data acquisition and preprocessing module of the present invention.

[0032] Figure 6 This is an internal block diagram of the intelligent hierarchical storage module of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1

[0034] As shown in the figure, this embodiment of the invention provides an optimization method for intelligent cloud data management, specifically including the following steps:

[0035] S1: Data Acquisition and Preprocessing: The data acquisition unit collects multi-source heterogeneous data from the cloud platform, including business data, user data, and device data. This includes structured data from a MySQL (Relational Database Management System) database, semi-structured data from Excel spreadsheets, and unstructured data such as images and videos. The preprocessing unit deduplicates the collected data, removing duplicate entries. Wavelet transform algorithms are used for noise reduction to eliminate data interference. Unstructured and semi-structured data are converted into JSON (JavaScript Object Notation) format structured data. Missing values ​​are filled using interpolation based on a normal distribution to ensure data integrity. The heat labeling unit uses PEBS (Performance Event Based) sampling. The sampling technology collects data access characteristics in real time. Combining access frequency (number of accesses per unit time), access duration (duration of a single access), and update frequency (number of updates per unit time), a weighted summation method is used to quantify data popularity. The weights are set to 0.4, 0.3, and 0.3, respectively. Popularity values ​​≥80 are considered hot data, 30 ≤ popularity value < 80 are considered warm data, and popularity values ​​< 30 are considered cold data. The encryption unit uses the AES-256 (Advanced Encryption Standard 256-bit) encryption algorithm to encrypt sensitive data (such as user privacy data and core business data).

[0036] S2: Intelligent Tiered Storage Optimization: The intelligent tiered storage module allocates storage nodes according to the data's popularity level. Hot data is stored in high-speed storage nodes composed of SSD solid-state drive arrays, with read / write speeds ≥1000MB / s; warm data is stored in conventional storage nodes composed of SAS hard disk storage arrays, with read / write speeds ≥300MB / s; cold data is migrated to archive storage nodes that combine tape libraries and cloud archive storage, with storage costs only 1 / 10 of those of high-speed storage nodes.

[0037] The decision monitoring cycle is set to 12 hours. The dynamic migration unit updates the data popularity level every 12 hours. Incremental migration is adopted, and only data whose popularity level has changed is migrated. The migration process adopts breakpoint resume technology. When the migration is interrupted, it can be restarted from the breakpoint to avoid data loss. The migration operation is performed from 2:00 to 4:00 am (the system low load period) to ensure that normal data access is not affected.

[0038] S3: Metadata Load Balancing Optimization: The metadata load balancing module constructs a machine learning-driven metadata management framework. The data collection unit collects metadata access load data in real time, including parameters such as metadata access frequency, access latency, CPU utilization, and memory usage for each node. The model training unit uses the gradient boosting tree algorithm to train a load prediction model based on historical access data from the past 30 days to predict the future load of each node. The decision-making unit dynamically allocates metadata to storage nodes with lower loads based on the prediction results, metadata locality characteristics, and hierarchical structure, avoiding access hotspots and keeping the load difference between nodes within 10%, reducing metadata access latency to below 50ms.

[0039] S4: Data Access Optimization: The data access optimization module constructs an intelligent caching mechanism, caching hot data and metadata accessed ≥100 times within the past 24 hours to edge nodes. Edge nodes are closer to users, reducing data transmission distance. Combined with the eBPF (Extended Berkeley Packet Filter) mechanism, the caching strategy is adjusted, and data access characteristics are monitored in real time. When the access frequency of a certain type of data decreases, it is automatically removed from the cache, improving the cache hit rate, which is maintained above 85%. At the same time, the data transmission protocol is optimized, adopting adaptive compression coding technology. Text data uses the LZ77 (Lempel-Ziv 77) algorithm, and binary data uses the Huffman coding algorithm, with the compression ratio controlled at about 1:4, reducing transmission bandwidth usage and increasing data transmission rate by more than 35%.

[0040] S5: Real-time Monitoring and Dynamic Adjustment: The monitoring unit collects system operating parameters in real time, monitors data storage status, access efficiency, and system load, and sets threshold warning mechanisms, including storage utilization (preset threshold 85%), data access latency (preset threshold 100ms), CPU utilization (preset threshold 75%), and memory usage (preset threshold 80%). When a parameter exceeds the preset threshold, the warning unit issues an audible and visual warning signal. The control unit automatically adjusts optimization strategies; for example, when storage utilization exceeds the threshold, some warm data is migrated to archive storage nodes; when access latency exceeds the threshold, the cache size of edge nodes is increased; when CPU utilization exceeds the threshold, non-urgent data migration operations are suspended to ensure stable system operation. Example 2

[0041] As shown in the figure, this embodiment of the invention provides an optimization system for intelligent cloud data management, specifically including a data acquisition and preprocessing module, an intelligent hierarchical storage module connected to the data acquisition and preprocessing module, a metadata load balancing module connected to the intelligent hierarchical storage module, a data access optimization module connected to the intelligent hierarchical storage module and the metadata load balancing module respectively, a real-time monitoring and control module, and a user interaction module for receiving parameter setting instructions input by the user. The real-time monitoring and control module is connected to the data acquisition and preprocessing module, the intelligent hierarchical storage module, the metadata load balancing module, the data access optimization module, and the user interaction module. All modules are connected via a bus and work together.

[0042] The data acquisition and preprocessing module includes a data acquisition unit, a preprocessing unit, a heat labeling unit, and an encryption unit. The data acquisition unit adopts a multi-interface acquisition method, supporting multiple interfaces such as MySQL (relational database management system), Hadoop (distributed computing framework), and FTP (File Transfer Protocol) to achieve comprehensive acquisition of multi-source heterogeneous data. The preprocessing unit adopts a distributed processing architecture to improve data preprocessing efficiency. The heat labeling unit integrates a PEBS (Performance Event Based Sampling) acquisition module to collect data access characteristics in real time. The encryption unit adopts hardware encryption to improve encryption efficiency and security.

[0043] The intelligent tiered storage module includes high-speed storage nodes, regular storage nodes, archive storage nodes, and a dynamic migration unit. High-speed storage nodes utilize SSD (Solid State Drive) arrays, numbering 8-16, forming a RAID5 (Redundant Array of Independent Disks Level 5) array to ensure data reliability. Regular storage nodes employ SAS (Serial Attached SCSI) hard drive arrays, numbering 16-32. Archive storage nodes combine tape libraries with cloud archive storage to support long-term storage of massive amounts of cold data. The dynamic migration unit integrates a breakpoint resume module to achieve secure and efficient data migration.

[0044] The metadata load balancing module includes a data collection unit, a model training unit, and a decision-making unit. The data collection unit collects metadata access load data from each node in real time and transmits it to the model training unit. The model training unit uses the gradient boosting tree algorithm to train a load prediction model based on historical data and updates the model parameters periodically. The decision-making unit dynamically allocates metadata storage nodes according to the model prediction results to achieve load balancing.

[0045] The data access optimization module includes a caching unit and a transmission optimization unit. The caching unit adopts a distributed caching architecture, supporting collaboration between edge nodes and cloud center caching. The transmission optimization unit integrates an adaptive compression encoding module, which automatically matches the data type and selects the optimal compression method.

[0046] The real-time monitoring and control module includes a monitoring unit, an early warning unit, and a control unit. The monitoring unit uses a real-time sampling method with a sampling frequency of 1 time / second to ensure the timeliness of parameter acquisition. The early warning unit supports audible and visual warnings as well as SMS warnings to promptly notify management personnel. The control unit uses an intelligent decision-making algorithm to automatically adjust and optimize strategies, while also supporting a manual control interface.

[0047] The user interaction module uses a web interface, which allows users to set optimization parameters (such as decision monitoring cycle, threshold parameters, etc.), view system operating status (storage utilization, access latency, etc.), and manually adjust optimization strategies. The operation is simple and convenient.

[0048] The system in this embodiment optimizes the entire cloud data management process through the collaborative work of its various modules during runtime. This effectively reduces storage costs, improves access efficiency, ensures stable system operation, and is adaptable to various cloud data management scenarios.

[0049] In summary, compared with the prior art, the present invention has the following advantages:

[0050] Achieve end-to-end optimization, covering all aspects of data collection, storage, access, and monitoring, and improve the overall efficiency of cloud data management through multi-stage optimization;

[0051] To reduce storage costs, intelligent tiered storage allocates data of varying popularity to storage nodes with corresponding costs, reducing waste of high-speed storage resources and lowering the cost of storing cold data. Compared to traditional storage methods, storage costs can be significantly reduced.

[0052] To improve access efficiency, metadata load balancing, intelligent caching, and transmission optimization can be used to reduce data access latency and increase cache hit rate, thereby effectively improving data access efficiency.

[0053] To ensure system stability, a real-time monitoring and dynamic control mechanism can promptly detect and handle system anomalies, automatically adjust and optimize strategies, avoid problems such as excessive system load and data loss, and improve system stability and reliability.

[0054] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. An optimized way for intelligent cloud data management, characterized in that: Includes the following steps: S1: Data Acquisition and Preprocessing: Collect heterogeneous data from multiple sources in the cloud platform, perform deduplication, noise reduction, and format standardization on the data to generate structured data; Data access characteristics are collected in real time using PEBS (Performance Event Sampling) technology, and data popularity levels are marked by data attributes, which are divided into three categories: hot data, warm data, and cold data. S2: Intelligent Tiered Storage Optimization: Based on data popularity level and access frequency, hot data is stored in high-speed cloud storage nodes, warm data is stored in regular cloud storage nodes, and cold data is migrated to low-cost archive storage nodes. A dynamic migration mechanism is established, and data popularity level is updated and storage location is adjusted every preset decision monitoring cycle. S3: Metadata load balancing optimization: Build a machine learning-driven metadata management framework to collect metadata access load data in real time. Based on the locality characteristics and hierarchical structure of metadata, dynamically allocate metadata storage nodes to avoid access hotspots and reduce metadata operation latency. S4: Data Access Optimization: Establish an intelligent caching mechanism to cache frequently accessed hot data and metadata to edge nodes, and adjust the caching strategy in conjunction with the eBPF (Extended Berkeley Packet Filter) mechanism to reduce the data transmission pressure in the cloud center; at the same time, optimize the data transmission protocol and use compression coding technology to reduce the transmission bandwidth consumption; S5: Real-time monitoring and dynamic adjustment: Real-time monitoring of data storage status, access efficiency, and system load; setting threshold early warning mechanisms; and automatically adjusting storage node allocation, caching strategies, and data migration schemes when an anomaly is detected to ensure stable and efficient system operation.

2. The optimization method for intelligent cloud data management of claim 1, wherein: In step S1, the data preprocessing also includes missing value imputation, which uses an interpolation method based on data distribution to fill missing data; the data popularity level is comprehensively quantified and evaluated by access frequency, access duration and data update frequency, and the weight of the quantification is dynamically adjusted according to user needs.

3. The optimization method for intelligent cloud data management of claim 1, wherein: In step S2, the dynamic migration mechanism adopts an incremental migration method, which only migrates data whose popularity level has changed. During the migration process, breakpoint resume technology is used to avoid data loss, and the migration operation is performed during periods of low system load, so as not to affect normal data access.

4. The optimization method for intelligent cloud data management of claim 1, wherein: In step S3, the machine learning-driven metadata management framework includes a data collection unit, a model training unit, and a decision unit. The model training unit uses the gradient boosting tree algorithm to train a load prediction model based on historical access data. The decision unit realizes metadata load balancing distribution based on the prediction results.

5. The optimization method for intelligent cloud data management of claim 1, wherein: In step S4, the compression coding technology adopts an adaptive compression algorithm, which automatically selects the optimal compression method according to the data type. The LZ77 algorithm is used for text data, and the Huffman coding algorithm is used for binary data. The compression ratio is controlled between 1:3 and 1:

5.

6. An optimization system for intelligent cloud data management, characterized by, include: Data acquisition and preprocessing module: used to collect multi-source heterogeneous data from the cloud platform, perform deduplication, noise reduction, format standardization, missing value imputation, label data popularity level, and output structured data; Intelligent hierarchical storage module: connected to the data acquisition and preprocessing module, it allocates storage nodes based on the data popularity level, realizing dynamic migration and hierarchical storage of hot, warm and cold data, including high-speed storage nodes, regular storage nodes and archive storage nodes; Metadata load balancing module: Connected to the intelligent tiered storage module, it is used to collect metadata access load data and realize the dynamic allocation of metadata storage nodes through machine learning models to optimize metadata access efficiency; Data access optimization module: It connects to the intelligent hierarchical storage module and the metadata load balancing module respectively, builds an intelligent caching mechanism, optimizes the data transmission protocol, and adopts compression encoding technology to improve data access and transmission efficiency; Real-time monitoring and control module: Connected to the data acquisition and preprocessing module, intelligent hierarchical storage module, metadata load balancing module, and data access optimization module, it is used to collect system operating parameters, data storage status, and access data in real time, set threshold warnings, automatically adjust system optimization strategies, and ensure stable system operation.

7. The optimization system for intelligent cloud data management of claim 6, wherein: The high-speed storage node uses an SSD solid-state drive array, the conventional storage node uses a SAS hard disk storage array, and the archive storage node uses a combination of tape library and cloud archive storage to reduce storage costs.

8. An optimization system for intelligent cloud data management according to claim 6, characterized in that: The real-time monitoring and control module includes a monitoring unit, an early warning unit, and a control unit. The monitoring unit collects parameters such as storage utilization, data access latency, and system load in real time. The early warning unit issues an early warning signal when the parameters exceed a preset threshold. The control unit automatically adjusts storage allocation and optimization strategies.

9. An optimization system for intelligent cloud data management according to claim 6, characterized in that: The data acquisition and preprocessing module also includes a data encryption unit, which uses the AES-256 encryption algorithm to encrypt the acquired sensitive data to ensure the security of data transmission and storage.

10. An optimization system for intelligent cloud data management according to claim 6, characterized in that: It also includes a user interaction module, which receives parameter setting instructions from users, displays the system's operating status and data optimization results, and supports users to manually adjust optimization strategies.

Images