Cloud multi-modal data dynamic archiving system for big data

Abstract

The invention relates to the technical field of data archiving, and comprises a big-data-oriented cloud multi-modal data dynamic archiving system which comprises a storage topology monitoring module, a storage hierarchy dynamic adjustment module, a data writing task decomposition module, a data consistency verification module and an archiving task dynamic execution module. According to the method, the storage node state is monitored, the storage topology is optimized, the data storage balance and the system throughput capacity are improved, the storage hierarchy is dynamically adjusted, the data write-in task distribution is performed according to the data access frequency, the storage resource utilization rate is improved, the computing resources of the adaptive storage nodes, the storage hot spots are avoided, and the real-time data consistency verification is utilized; storage abnormity is found and corrected in time, data reliability is guaranteed, storage node network time delay and load conditions are considered in an archiving task, data storage distribution is optimized, archiving efficiency and system stability are improved, data processing efficiency is improved jointly, and high availability and expandability are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of data archiving, and in particular to a cloud-based multi-modal data dynamic archiving system for big data. Background Art

[0002] The technical field of data archiving includes methods for long-term storage, management, and retrieval of digital information. The core contents include data collection, classification, storage, migration, and recovery to ensure the availability and integrity of data over a long period. Data archiving usually involves the conversion of different storage media, such as transferring frequently accessed data from high-speed storage devices to low-cost long-term storage media. This field also covers means such as data compression, deduplication, and index management to optimize storage space and improve retrieval efficiency. In addition, with the development of cloud computing and big data technologies, data archiving is gradually evolving towards a cloud-based distributed storage architecture, enabling dynamic management and access of data among multiple nodes, and improving the flexibility and scalability of the archiving system.

[0003] Among them, a cloud-based multi-modal data dynamic archiving system for big data refers to a method for dynamically managing and archiving different types of data in a cloud computing environment, covering the classification and storage of various data types, including structured data, semi-structured data, and unstructured data, and using data parsing and feature extraction methods to classify and archive data. During the data storage process, based on a distributed storage architecture, data is stored in chunks, and a cross-node storage strategy is used to achieve load balancing and high availability of data. In addition, a scheduling mechanism based on timestamps and access frequencies is adopted to achieve automatic migration and lifecycle management of data, and a consistent hashing algorithm is combined to optimize the data allocation strategy to improve the efficiency and reliability of data access.

[0004] During the existing cloud-based multi-modal data archiving process, there is a lack of dynamic monitoring of the status of storage nodes in storage topology management, making it difficult to accurately evaluate the load conditions of storage nodes, resulting in overloading of some storage nodes and affecting the stability of data archiving. The storage hierarchy adjustment method relies on fixed policies and fails to optimize the storage hierarchy according to changes in access frequencies, causing frequently accessed data to be stored in low-priority levels and affecting data retrieval performance. Storage task allocation does not fully consider the computing resource occupancy of storage nodes, resulting in overloading of tasks on some storage nodes, affecting the data writing rate, and increasing the overall burden on the storage system. Data consistency verification mainly relies on periodic offline detection, making it difficult to discover consistency problems in real time during the storage process, and easily leading to undetected data corruption, affecting the integrity of archived data. During the execution of archiving tasks, the network transmission delay and load status of storage nodes are not fully considered, resulting in high-load nodes still being assigned storage tasks, reducing the execution efficiency of data archiving, and increasing the instability of the storage system. Summary of the Invention

[0005] The object of the present invention is to solve the disadvantages existing in the prior art, and a cloud multi-modal data dynamic archiving system for big data is proposed.

[0006] To achieve the above object, the present invention adopts the following technical solutions: The cloud multi-modal data dynamic archiving system for big data includes: The storage topology monitoring module obtains the connection status, bandwidth occupancy rate and I / O load of the storage nodes, calls the storage topology mapping table, compares the topology connection density of each storage node, calculates the topology weight of the storage nodes, and generates storage topology status data; The storage level dynamic adjustment module, based on the storage topology status data, counts the access times of multi-modal data within a set time window, calculates the access frequency threshold interval of the data, compares the storage level access priority reference value, analyzes the matching degree between the data storage level and the access frequency, determines the target position of the data level adjustment, and generates a storage level adjustment plan; The data writing task decomposition module, based on the storage level adjustment plan, obtains the data block size of the data to be archived, combines the computing resources of the storage nodes, filters the storage nodes with the computing resource occupancy rate lower than the set resource value, calculates the data block storage task allocation ratio according to the filtering result, and generates a writing task decomposition plan; The data consistency verification module applies the writing task decomposition plan, obtains the write confirmation status of the storage nodes, compares the data consistency between the storage nodes, calculates the data consistency deviation value, and generates a data consistency verification result.

[0007] As a further solution of the present invention, the storage topology status data includes the topology connection density of the storage nodes, the topology weight of the storage nodes, and the bandwidth utilization rate of the storage nodes. The storage level adjustment plan includes the data access frequency threshold interval, the storage level access priority reference value, and the target position of the data level adjustment. The writing task decomposition plan includes the data block storage task allocation ratio, the filtering result of the computing resource occupancy rate of the storage nodes, and the list of target storage nodes. The data consistency verification result includes the write confirmation status of the storage nodes, the data consistency deviation value between the storage nodes, and the data integrity verification result.

[0008] As a further solution of the present invention, the storage topology monitoring module includes: The connection status detection sub-module obtains the connection status data of the storage nodes, collects the connection information between the storage nodes, including the port status, connection validity and transmission stability of the nodes, judges the connection validity of each storage node according to the port connection success rate and connection delay parameters of the nodes, and calculates the connection stability data of each storage node; Based on the connection stability data of the storage nodes, the bandwidth and I / O load analysis sub-module collects the bandwidth occupancy and I / O load data of the storage nodes, obtains the current bandwidth utilization rate, I / O request rate and queue depth of the storage nodes, calculates the balance between bandwidth occupancy and I / O load, and obtains the bandwidth-I / O balance degree data of each storage node; According to the bandwidth-I / O balance degree data, the topology weight calculation sub-module obtains the topology connection density of each storage node based on the topology mapping table of the storage topology, calculates the comprehensive influence value of the topology connection number of the storage node and its bandwidth-I / O balance degree, and uses the formula: ; Calculate the topology weight of the storage node and generate the storage topology state data in combination with the topology structure information of the storage node. Among them, represents the topology connection number between storage node and node , represents the bandwidth-I / O balance degree value of storage node , represents the connection stability of storage node , and represents the total number of storage nodes in the storage topology.

[0009] As a further solution of the present invention, the storage hierarchy dynamic adjustment module includes: Based on the storage topology state data, the access frequency statistics sub-module obtains the number of accesses of multi-modal data within a set time window, counts the access count values of each data block, calculates the access frequency of each data block, and calculates the access frequency change trend according to the time window to obtain the access frequency trend data of each data block; Based on the access frequency trend data, the storage hierarchy matching analysis sub-module calculates the access frequency threshold interval of the data, compares it with the access priority reference value of the storage hierarchy, determines whether the access frequency of each data block meets the access reference range corresponding to the storage hierarchy, calculates the matching degree between the data storage hierarchy and the access frequency, and obtains the storage matching degree data; According to the storage matching degree data, the hierarchy adjustment target calculation sub-module determines the target position of the hierarchy adjustment of the data based on the matching degree between the data storage hierarchy and the access frequency, and uses the formula: ; Calculate the adjusted target hierarchy value of data block and generate the storage hierarchy adjustment plan in combination with the storage topology state data. Among them, represents data block ​​The access frequency trend value, represents the storage hierarchy The access priority reference value, represents the data block The current storage hierarchy stability of, represents the total number of storage hierarchies in the storage system.

[0010] As a further solution of the present invention, the data writing task decomposition module includes: The storage hierarchy data block acquisition sub-module determines the data archiving rules for different storage hierarchies based on the storage hierarchy adjustment scheme, analyzes the current data occupancy of the storage device, extracts the data blocks that meet the archiving criteria, calculates the total amount and distribution ratio of the data blocks under each storage hierarchy according to the distribution of the data blocks, and obtains the data block distribution data of each storage hierarchy; The storage node computing resource screening sub-module screens all storage nodes based on the data block distribution data of each storage hierarchy, combines the computing resource occupancy information of the storage nodes, sets a threshold according to the computing resource utilization rate, extracts the storage nodes with a computing resource occupancy rate lower than the threshold, and obtains the storage nodes available for task allocation; The storage task allocation ratio calculation sub-module calculates the available storage ratio of each storage node based on the storage nodes available for task allocation and the data block distribution data of each storage hierarchy, combines the target data block size of the storage task, and uses the formula: ; Calculate the storage node The storage task allocation ratio of to generate a writing task decomposition scheme, where represents the storage node in the storage hierarchy The total amount of data blocks of, represents the storage node The remaining computing resources of, represents the total remaining computing resources of the available storage nodes, represents the total data block size of the storage task, represents the total number of storage nodes, represents the number of available storage nodes.

[0011] As a further solution of the present invention, the data consistency verification module includes: The storage node write confirmation sub-module applies the write task decomposition scheme, obtains the write confirmation status of all storage nodes, counts the number of successful writes and the number of failed writes of each storage node, and calculates the write success rate of the storage node; The data consistency deviation calculation sub-module calculates the hash difference between storage nodes by comparing the data hash values stored in different storage nodes based on the write success rate of the storage nodes, and combines the write timestamp difference of the storage blocks. Using the formula: ; Calculate the data consistency deviation value , where represents the data hash value of the storage node , represents the data hash value of the reference storage node, represents the storage node 's data write timestamp, represents the write timestamp of the reference storage node, represents the total number of storage nodes; The consistency verification sub-module compares the data consistency deviation value with a set consistency threshold to determine whether the data meets the consistency standard and obtains the data consistency verification result.

[0012] As a further solution of the present invention, the system further includes an archiving task dynamic execution module; The archiving task dynamic execution module obtains the network transmission delay and load status data of the storage nodes based on the data consistency verification result, filters out the storage nodes with a load status lower than the set load value, determines the target node for migrating the write task of the data block, and executes the storage archiving operation of the data block.

[0013] As a further solution of the present invention, the archiving task dynamic execution module includes: The storage node status monitoring sub-module obtains the network transmission delay and load status data of the storage nodes based on the data consistency verification result, statistically calculates the real-time load occupancy rate of each storage node, calculates the average transmission delay of each storage node, and obtains the network transmission load status of the storage nodes; The write task migration target screening sub-module filters out the storage nodes with a load occupancy rate lower than the set load value according to the network transmission load status of the storage nodes, calculates the load balancing situation of each storage node, analyzes the influence degree of the transmission delay, and combines the available resource situation of the storage nodes to determine the storage nodes suitable as the target for migrating the write task of the data block, and obtains the target node for migrating the write task of the data block; The data block storage archiving execution sub-module executes the storage archiving operation of the data block based on the target node for migrating the write task of the data block, calls the storage node to transfer and archive the write task, and completes the cloud multi-modal data archiving.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In the present invention, by monitoring the connection status, bandwidth occupancy rate, and I / O load of storage nodes, the storage topology structure is optimized to improve the balance of data storage, reduce the congestion risk of high-load nodes, enhance the throughput capacity of the archival system, adjust the storage hierarchy based on dynamic statistics of access frequencies to ensure that frequently accessed data is stored in a storage layer with a higher priority, migrate low-frequency data to more cost-effective storage media, optimize the utilization rate of storage resources, reduce unnecessary data migration overheads, combine the computing resource occupancy of storage nodes, screen suitable storage nodes for data write task allocation, reduce storage hotspot problems, improve the scheduling efficiency of storage tasks, adopt a method of dynamically matching computing resource screening and storage task allocation ratio for data block storage tasks to balance the storage task load, enhance the data write efficiency, and reduce storage bottlenecks caused by overloading of storage nodes. During the data consistency verification process, real-time data integrity detection is achieved by combining the write confirmation status of storage nodes and calculation of data consistency deviations, promptly detect storage anomalies, and ensure the reliability of archival data. When the archival task is executed, target storage nodes are screened based on the network transmission delay and load conditions of storage nodes to avoid high-load nodes from affecting the execution efficiency of the archival task, optimize the global distribution of data storage, and enhance stability and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is the system flowchart of the present invention; Figure 2 is the flowchart of the storage topology monitoring module of the present invention; Figure 3 is the flowchart of the storage hierarchy dynamic adjustment module of the present invention; Figure 4 is the flowchart of the data write task decomposition module of the present invention; Figure 5 is the flowchart of the data consistency verification module of the present invention; Figure 6 is the flowchart of the archival task dynamic execution module of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] In order to make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0017] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention. In addition, in the description of the present invention, the meaning of "a plurality of" is two or more, unless otherwise specifically defined.

[0018] Please refer to Figure 1 , the cloud multi-modal data dynamic archiving system for big data includes: The storage topology monitoring module obtains the connection status, bandwidth occupancy rate, and I / O load of storage nodes, calls the topology mapping table of the storage topology, compares the topology connection density of each storage node, calculates the topology weight of the storage node, and generates storage topology status data; The storage hierarchy dynamic adjustment module, based on the storage topology status data, counts the number of accesses of multi-modal data within a set time window, calculates the access frequency threshold interval of the data, compares it with the access priority reference value of the storage hierarchy, analyzes the matching degree between the data storage hierarchy and the access frequency, determines the target position of the hierarchical adjustment of the data according to the matching degree, and generates a storage hierarchy adjustment plan; The data writing task decomposition module, based on the storage hierarchy adjustment plan, obtains the data block size of the data to be archived, combines the computing resources of the storage nodes, filters out the storage nodes with a computing resource occupancy rate lower than the set resource value, calculates the storage task allocation ratio of the data block according to the filtering result, and generates a writing task decomposition plan; The data consistency verification module applies the writing task decomposition plan, obtains the write confirmation status of the storage nodes, compares the data consistency between the storage nodes, calculates the data consistency deviation value, and generates a data consistency verification result; The archiving task dynamic execution module, based on the data consistency verification result, obtains the network transmission delay and load status data of the storage nodes, filters out the storage nodes with a load status lower than the set load value, determines the target node for migrating the write task of the data block, and executes the storage archiving operation of the data block.

[0019] The stored topology status data includes the topology connection density of storage nodes, the topology weight of storage nodes, and the bandwidth utilization rate of storage nodes. The storage hierarchy adjustment scheme includes the data access frequency threshold range, the storage hierarchy access priority reference value, and the target location for data hierarchy adjustment. The write task decomposition scheme includes the data block storage task allocation ratio, the screening result of the computing resource occupancy rate of storage nodes, and the list of target storage nodes. The data consistency verification result includes the write confirmation status of storage nodes, the data consistency deviation value between storage nodes, and the data integrity verification result.

[0020] Please refer to Figure 2 , the storage topology monitoring module includes: The connection status detection sub-module obtains the connection status data of storage nodes, collects the connection information between storage nodes, including the port status, connection validity, and transmission stability of nodes, judges the connection validity of each storage node based on the port connection success rate and connection delay parameters of the nodes, and calculates the connection stability data of each storage node; Obtain the connection status data of storage nodes. The connection information between storage nodes includes port status, connection validity, and transmission stability. To analyze the connection stability of storage nodes, it is necessary to monitor the connection validity of ports and calculate the average effective connection number and fluctuation standard deviation within a certain time interval. . In the actual implementation process, for 3 ports of a certain storage node, 10 samples are taken within a monitoring period of the past 60 seconds, and the effective connection number of the port is recorded each time.

[0021] Table 1 Port connection validity record (unit: times) As shown in Table 1, the connection situation of each port fluctuates slightly. To calculate the connection stability of the port, first calculate the connection stability standard deviation of each port .

[0022] Taking port 1 as an example, its mean value is 54.7, and calculate its standard deviation: ; ; Similarly, calculate the standard deviations of ports 2 and 3, and get: ; Then, calculate the connection stability by normalization: ; Among them, (take the standard deviation of port P3), substitute it into the calculation: ; ; ; This result shows that the connection stability of Port 2 is the highest, the connection fluctuation of Port 3 is the largest, and the connection state is the most unstable.

[0023] Based on the connection stability data of the storage nodes, the bandwidth and I / O load analysis sub-module collects the bandwidth occupancy and I / O load data of the storage nodes, obtains the current bandwidth utilization rate, I / O request rate, and queue depth of the storage nodes, calculates the balance between bandwidth occupancy and I / O load, and obtains the bandwidth-I / O balance degree data of each storage node.

[0024] Based on the connection stability data of the storage nodes, the bandwidth and I / O load analysis sub-module collects the bandwidth occupancy and I / O load data of the storage nodes, obtains the current bandwidth utilization rate, I / O request rate, and queue depth of the storage nodes, calculates the balance between bandwidth occupancy and I / O load, and obtains the bandwidth-I / O balance degree data of each storage node; Based on the connection stability data of the storage nodes, the bandwidth occupancy and I / O load data of the storage nodes are collected. It is necessary to monitor the bandwidth utilization rate and I / O rate of each storage node, calculate its fluctuation in different time periods to measure its balance degree. In order to analyze the I / O load and bandwidth usage more accurately, we monitor the I / O rate and queue depth in different time periods, and the data is as follows: Table 2 I / O Load Monitoring Data The bandwidth-I / O balance degree is calculated as follows: ; Substitute the data in Table 2 for calculation: ; ; ; The calculated bandwidth-I / O balance degree value is 20110. This value represents the average balance degree of the I / O load of this storage node in 5 time periods. Numerically, the I / O request volume of this storage node is relatively large during this period, and the relationship between its queue depth and request rate is relatively stable, which indicates that the I / O resource utilization rate of this storage node is relatively balanced. In practical applications, if the bandwidth-I / O balance degree value is too low, it may indicate insufficient I / O resource utilization of the storage device, affecting the storage throughput; if the bandwidth-I / O balance degree value is too high, it may indicate that the I / O resources of the storage device are in an overloaded state, which may affect the data read and write performance.

[0025] Based on the bandwidth - I / O balance data and according to the topology mapping table of the storage topology, the topology weight calculation sub - module obtains the topology connection density of each storage node, calculates the comprehensive influence value of the topology connection number of the storage node and its bandwidth - I / O balance degree, and uses the formula: ; Calculate the topology weight of the storage node Combined with the topology structure information of the storage node, generate storage topology status data, where represents the topology connection number between storage node and node , represents the bandwidth - I / O balance degree value of storage node , represents the connection stability of storage node , ; represents the total number of storage nodes in the storage topology; Based on the bandwidth - I / O balance data and according to the topology mapping table of the storage topology, obtain the topology connection density of each storage node, calculate the comprehensive influence value of the topology connection number of the storage node and its bandwidth - I / O balance degree. Suppose there are 5 storage nodes in the storage topology, and the connection density , bandwidth - I / O balance degree and connection stability monitoring values are as follows: Table 3 Storage Node Topology Parameters As shown in Table 3, substitute into the formula to calculate the topology weight of storage node : ; ; The result shows that the topology weight of storage node is 7.18, and the storage topology status data can be used for subsequent optimization of the topology connection scheme.

[0026] Please refer to Figure 3 , the storage hierarchy dynamic adjustment module includes: The access frequency statistics sub - module, based on the storage topology status data, obtains the access times of multi - modal data within a set time window, counts the access count values of each data block, calculates the access frequency of each data block, and calculates the access frequency change trend according to the time window to obtain the access frequency trend data of each data block; Obtain the access count of multimodal data within a set time window, count the access count values of each data block, calculate the access frequency of each data block, and calculate the access frequency change trend based on the time window to obtain the access frequency trend value of each data block. For multimodal data, the set time window can be 1 hour, 1 day, or 1 week, depending on the application scenario, such as database storage, high-performance computing storage, or distributed storage systems. In a storage system, the access count can be completed through log analysis and I / O request records. Assume that the data access logs received by a certain storage node within 24 hours are as follows: Table 4 Access Log Data As shown in Table 4, the access counts of different data blocks are different at different time periods. To calculate the access frequency, set the access time window to 6 hours, and the calculation method is as follows: Access Frequency = Access Count ÷ Time Window For example, the access frequency of data block A1 = 150 ÷ 6 = 25 times / hour Similarly, calculate A2, A3, and A4: A2: 230 ÷ 6 = 38.3 times / hour A3: 180 ÷ 6 = 30 times / hour A4: 200 ÷ 6 = 33.3 times / hour When calculating the access frequency change trend, a sliding window calculation method can be used. For example, calculate the change trend between A1 and A2: ; After calculating the trend values of other data blocks in sequence, the final access frequency trend data can be obtained.

[0027] The storage level matching analysis sub-module calculates the access frequency threshold interval of the data based on the access frequency trend data, compares it with the access priority benchmark value of the storage level, determines whether the access frequency of each data block meets the access benchmark range corresponding to the storage level, calculates the matching degree between the data storage level and the access frequency, and obtains the storage matching degree data; Based on the access frequency trend data, calculate the access frequency threshold interval of the data, compare it with the access priority benchmark value of the storage level, determine whether the access frequency of each data block meets the access benchmark range corresponding to the storage level, calculate the matching degree between the data storage level and the access frequency, and obtain the storage matching degree data. Assume that there are three levels in the storage system: Level 1: High-priority storage (SSD); Level 2: Medium-priority storage (HDD); Level 3: Low-priority storage (cold storage); Define the access priority benchmark values of each storage level as follows: Level 1: ≥ 30 times per hour; Level 2: 10 - 30 times per hour; Level 3: ≤ 10 times per hour; As shown in Table 5, compare the aforementioned access frequency with the level priority benchmark value to determine the matching degree.

[0028] Table 5 stores the matching degree calculation Formula for calculating the matching degree: ; For example, the matching degree calculation of A3 (here take the upper limit value of the level as the calculation standard, that is, 30): ; A matching degree equal to 1 indicates that data block A3 matches the current level.

[0029] The level adjustment target calculation sub-module determines the level adjustment target position of the data according to the stored matching degree data and the matching degree between the data storage level and the access frequency, using the formula: ; Calculate the adjustment target level value of data block , and combine it with the storage topology state data to generate a storage level adjustment plan, where represents the access frequency trend value of data block , represents the access priority benchmark value of storage level , represents the current storage level stability of data block , and represents the total number of storage levels in the storage system; Call the stored matching degree data, determine the level adjustment target position of the data according to the matching degree between the data storage level and the access frequency, calculate the adjustment target level value of each data block using the formula, and combine it with the storage topology state data to generate a storage level adjustment plan.

[0030] For data block A2: Let (storage stability) = 0.9 (storage level 1) Calculate: ; ; ; ; Calculation Part Two: ; ; ; ; Compare Storage Tier Adjustment Target: If , the storage tier is adjusted to SSD; If , the storage tier is adjusted to HDD; If , the storage tier is adjusted to cold storage; This calculation shows that the Block A2 data block is suitable for the SSD storage tier. Therefore, the specific storage tier adjustment plan is as follows: Keep Block A1 on HDD; Migrate Block A2 to SSD; Keep Block A3 on HDD; Migrate Block A4 to SSD.

[0031] Please refer to Figure 4 , the data writing task decomposition module includes: The storage tier data block acquisition sub-module determines the data archiving rules for different storage tiers based on the storage tier adjustment plan, analyzes the current data occupancy of the storage device, extracts the data blocks that meet the archiving criteria, calculates the total amount and distribution ratio of the data blocks under each storage tier according to the distribution of the data blocks, and obtains the data block distribution data for each storage tier; The hierarchical structure information of a storage system usually includes different storage media such as cache (e.g., SSD), disk storage (e.g., HDD), and cold storage (e.g., tape storage). The access speed and storage cost of each medium are different. Therefore, the setting of data archiving rules needs to be based on the characteristics of the storage hierarchy. First, call the current data occupancy of the storage device, obtain the number of existing data blocks in each storage hierarchy, and determine the data block size distribution. Adopt a hierarchical traversal method to sequentially read the metadata of the storage system and extract the distribution status of data blocks under each storage hierarchy. For example, assume that the storage system consists of two levels of storage hierarchy. Among them, the cache layer stores 20TB of data, the disk storage layer stores 80TB of data, and the cold storage layer stores 150TB of data. Then the data ratios of each storage hierarchy are 11.8%, 47.1%, and 41.1% respectively. When calculating the data block distribution ratio of each storage hierarchy, assume that in the 20TB of data stored in the cache layer, the size of each data block is 4MB, then the number of data blocks is 5,000,000; in the disk storage layer, the data block size is 16MB, then the number of data blocks is 5,000,000; in the cold storage layer, the data block size is 64MB, then the number of data blocks is 2,343,750. Based on this, calculate the data block distribution ratio of each storage hierarchy through the proportion of the number of data blocks, as shown in Table 6.

[0032] Table 6 Data Block Distribution Ratio Table for Each Storage Hierarchy As shown in Table 6, the proportion of the number of data blocks in the disk storage layer is the same as that in the cache layer, but the data block sizes are different, and the number of data blocks in the cold storage layer is relatively small. Combine the access frequency of the storage hierarchy and the access mode of the data blocks to obtain the data block distribution data of each storage hierarchy.

[0033] Based on the data block distribution data of each storage hierarchy, the storage node computing resource screening sub-module combines the computing resource occupancy information of the storage nodes, sets a threshold according to the computing resource utilization rate, screens all storage nodes, extracts the storage nodes with a computing resource occupancy rate lower than the threshold, and obtains the storage nodes available for task allocation; Based on the data block distribution ratio of each storage level, the computing resource occupancy information of the storage nodes is called. The computing resource occupancy of each storage node includes CPU usage rate, memory usage rate, and I / O bandwidth occupancy rate. Assume that the total CPU resources of the storage node are 64 cores, and the CPU usage rates of each storage node at a certain moment are 25%, 40%, 55%, and 70% respectively. Set the computing resource utilization threshold to 50%. The setting basis of this threshold lies in the load balancing management principle of computing resources, that is, when the CPU usage rate is higher than 50%, the execution efficiency of computing tasks will be affected. According to the analysis of the historical operating status of the storage node, when the CPU usage rate exceeds 50%, the average response time of I / O access requests will increase to more than 1.5 times the original delay, and at the same time, the throughput of data writing will also decrease by more than 20%. Therefore, the setting of this threshold is related to the computing resource scheduling ability of the storage node and fluctuates with the changes in the CPU main frequency, memory bandwidth, and storage controller performance of the storage node. Filter out the storage nodes with CPU occupancy lower than this threshold, that is, select the storage nodes with CPU occupancy of 25% and 40%.

[0034] Then, combined with the I / O bandwidth occupancy of the storage nodes, assume that the total bandwidth of the storage node is 10 Gbps, and the bandwidth occupancies of each node are 3 Gbps, 5 Gbps, 6 Gbps, and 8 Gbps respectively. Set the bandwidth occupancy threshold to 60%. The setting of this threshold is based on the data transmission rate of the storage system and the service quality requirements of read and write requests. Usually, in the design of the storage system, after the bandwidth occupancy exceeds 60%, the packet loss rate will increase from 0.1% to more than 1%, resulting in a significant decrease in data access performance. At the same time, the processing ability for parallel data streams is reduced, and the queuing waiting time for write tasks increases, thus affecting the execution efficiency of the overall storage task. The adjustment range of this threshold is affected by the network bandwidth topology of the storage node, the packet scheduling strategy, and the multi-channel data stream control ability inside the storage node. According to the set threshold, filter out the storage nodes with bandwidth occupancy lower than 60%, that is, the storage nodes with bandwidth occupancy of 3 Gbps and 5 Gbps. After taking the intersection, finally determine the storage nodes available for task allocation, as shown in Table 7.

[0035] Table 7 Screening Table of Storage Nodes Available for Task Allocation As shown in Table 7, finally, storage nodes 1 and 2 with CPU usage rate lower than 50% and I / O bandwidth occupancy lower than 60% are screened out to obtain the storage nodes available for task allocation.

[0036] The storage task allocation ratio calculation sub-module, based on the storage nodes available for task allocation and the data block distribution data of each storage level, calculates the available storage ratio of each storage node, and combines it with the target data block size of the storage task, using the formula: ; Calculate the storage task allocation ratio of the storage nodes to generate a write task decomposition plan, where represents the total amount of data blocks of the storage nodes in the storage hierarchy, represents the remaining computing resources of the storage node , represents the total remaining computing resources of the available storage nodes, represents the total size of the storage task data blocks, represents the total number of storage nodes, represents the number of available storage nodes; Based on the storage nodes available for task allocation and the data block distribution ratio of each storage hierarchy, calculate the available storage ratio of each storage node. First, calculate the total amount of data blocks of storage nodes 1 and 2. Assume that storage node 1 stores 1,500,000 data blocks and storage node 2 stores 1,200,000 data blocks, and the total number of data blocks is 2,700,000. According to the target data block size of the storage task, assume that the total data block size of the task is 10TB.

[0037] Assume that the remaining computing resources of storage node 1 are 20 CPU cores and the remaining computing resources of storage node B are 18 CPU cores, then: ; ; Calculate the storage task allocation ratio of storage node 1: ; ; ; Calculate the storage task allocation ratio of storage node 2: ; ; ; Finally, the storage task allocation ratio is 4.03TB for storage node 1 and 3.06TB for storage node 2, thereby obtaining the storage task allocation ratio of each storage node and generating a write task decomposition plan.

[0038] Please refer to Figure 5 , the data consistency verification module includes: ​The storage node write confirmation sub-module applies the write task decomposition scheme, obtains the write confirmation status of all storage nodes, counts the number of successful writes and failed writes for each storage node, and calculates the write success rate of the storage nodes. Call the write task decomposition scheme. First, obtain the write confirmation status of all storage nodes, extract the log information of the storage nodes, analyze the timestamps, operation types, data identifiers, and confirmation response codes in the log records, classify and count the write situations of the storage nodes, distinguish between successful writes and failed writes, summarize the statistical data for different storage nodes, and calculate the write success rate of each storage node. Suppose a storage system contains 5 storage nodes, and the total number of write requests and the number of successful writes for each storage node are shown in Table 8.

[0039] Table 8 Storage Node Write Statistics Table As shown in Table 8, the calculation method of the write success rate is: ; Taking storage node 1 as an example, its write success rate is: ; Obtain the write success rate of the storage nodes.

[0040] Based on the write success rate of the storage nodes, compare the data hash values stored in different storage nodes, calculate the hash difference between the storage nodes, and combine the write timestamp difference of the storage blocks. Use the formula: ; Calculate the data consistency deviation value , where represents the data hash value of storage node , represents the data hash value of the reference storage node, represents the write timestamp of the data of storage node , represents the write timestamp of the reference storage node, represents the total number of storage nodes; Based on the write success rate of the storage nodes, compare the data hash values stored in different storage nodes, extract the stored data hash values of each storage node, calculate the hash difference between the storage nodes, and perform quantitative analysis in combination with the write timestamps of the storage blocks. Suppose there are 3 storage nodes storing the same data block, but their hash values are different and there are also certain differences in timestamps, as shown in Table 9.

[0041] Table 9 Storage Node Data Hash and Timestamp Table The consistency deviation value of the data is calculated using a formula, and the calculation is performed based on the data in Table 9: For storage node 1: ; For storage node 2: ; For storage node 3: ; The final consistency deviation value: ; Furthermore, the data consistency deviation value is obtained.

[0042] The consistency verification sub-module compares the data consistency deviation value with the set consistency threshold to determine whether the data meets the consistency standard and obtains the data consistency verification result; Based on the data consistency deviation value, it is compared with the set consistency threshold to determine whether the data meets the consistency standard. The setting of the data consistency threshold is based on the data replication mechanism of the storage system, the data synchronization rate of the storage nodes, and the error tolerance of the storage medium. Usually, the storage system comprehensively sets the consistency threshold based on the historical write success rate, hash matching rate, and write time deviation to ensure the synchronization stability of the data between different storage nodes. In a high-reliability storage environment, the write success rate usually remains above 95%, the hash matching rate is close to 100%, and the write time deviation is generally controlled within 3 seconds. Therefore, the setting of the consistency threshold is usually adjusted according to the fluctuation range of these three parameters. In this example, the set threshold is 5, that is, considering that the storage nodes may cause the hash deviation to accumulate to a certain extent due to factors such as network latency and I / O load imbalance, and 5 as the threshold can cover the common data storage consistency deviation range and avoid misjudgment caused by being too strict. If the calculated data consistency deviation value is less than or equal to 5, it is determined that the data is consistent, otherwise the data is inconsistent. Combining the previously calculated data consistency deviation value , this value exceeds the set threshold, so it is determined that the data is inconsistent. Obtain the data consistency verification result.

[0043] Please refer to Figure 6 , the archive task dynamic execution module includes: Based on the data consistency verification result, the storage node status monitoring sub-module obtains the network transmission delay and load status data of the storage nodes, counts the real-time load occupancy rate of each storage node, calculates the average transmission delay of each storage node, and obtains the network transmission load status of the storage nodes; Based on the data consistency verification results, first obtain the network transmission delay and load status data of each storage node from the storage system. Specifically, it is necessary to call the network monitoring logs of the storage nodes to extract the network transmission delay data of each storage node at different time periods. At the same time, collect the CPU occupancy rate, memory usage, and disk I / O load status of the storage nodes to evaluate the current load situation of the storage nodes. In actual operation, the CPU usage rate of each node can be obtained through the SNMP protocol of the monitoring system (for example, the CPU occupancy rate of a certain storage node is 30%), and calculations are performed in combination with I / O throughput data (such as the disk I / O throughput of a certain node is 150M2 / s). After statistically calculating the real-time load occupancy rate of each storage node, it is necessary to normalize the load situations of different resources. Different weights are set for the CPU occupancy rate, memory usage rate, and I / O throughput. The basis for setting the weights lies in the degree of influence of each resource on the performance of the storage task. The CPU occupancy rate directly affects the data processing ability, so the weight is set to 0.4. The memory usage rate affects the data caching and writing speed, so the weight is set to 0.3. And the I / O throughput determines the actual access speed of the data, so the weight is set to 0.3. This set value depends on the architecture characteristics of the storage system. For example, for compute-intensive tasks, the CPU weight can be increased, and for large-scale data transfer tasks, the I / O throughput weight should be appropriately increased. Then calculate the comprehensive load occupancy rate. For example, for a certain storage node, its comprehensive load can be expressed as: ; In addition, it is necessary to calculate the average network transmission delay of each storage node. Assume that the measured delays of a certain storage node in the past 1 minute are 20ms, 25ms, 22ms, 21ms, and 24ms respectively. Then its average network delay is calculated as follows: ; In this way, the network transmission load status of each storage node can be obtained, and the resource occupancy situation of the storage nodes can be evaluated to determine the subsequent task migration strategy, and finally the network transmission load status of the storage nodes is obtained.

[0044] The write task migration target screening sub-module screens the storage nodes with a load occupancy rate lower than the set load value according to the network transmission load status of the storage nodes, calculates the load balancing situation of each storage node, analyzes the degree of influence of the transmission delay, and combines the available resource situation of the storage nodes to determine the storage nodes suitable as the write task migration target, and obtains the write task migration target nodes of the data blocks; Screen the storage nodes according to the network transmission load status of the storage nodes. First, set a load threshold to exclude storage nodes with a load status higher than the set value. The setting of the load threshold is based on the processing capacity of the storage nodes, the historical average load level, and the scheduling strategy of the storage tasks. Usually, the load rate of the storage nodes is mainly affected by the 3PU occupancy rate, memory usage rate, and disk I / O throughput. When the load of the storage node exceeds a certain value, its task processing ability will gradually decline. To ensure the stability of data migration, the load threshold is usually set within the optimal range of the processing capacity of the storage nodes. For example, in most enterprise-level storage systems, when the load rate exceeds 60%, the response time of the storage tasks will increase significantly, while when it is below 30%, the storage resources are not fully utilized. Therefore, 50% is taken as the load threshold to make it in a state of balance between resource utilization and task scheduling. This value will be adjusted according to the hardware specifications of the storage nodes. For example, higher-performance storage nodes can accept a higher load rate, while the load threshold of older storage nodes should be appropriately reduced. Assuming the load threshold is set at 50%, then screen out the storage nodes with a load occupancy rate lower than 50%. For example, if the load rate of a certain storage node 1 is 38%, the load rate of storage node 2 is 47%, and the load rate of storage node 3 is 55%, then only storage nodes 1 and 2 meet the screening conditions for the migration target. Among the screened storage nodes, calculate the load balance of each storage node to avoid over-concentrated task allocation. The load balance index can be evaluated by calculating the current task proportion of the storage nodes. Assuming that the current number of storage tasks of storage node 1 is 100, accounting for 20%, the current number of tasks of storage node 2 is 150, accounting for 30%, and the number of tasks of node 3 is 250, accounting for 50%. Since the load of node 3 has exceeded the threshold, only 1 and 2 enter the migration target evaluation range. Then analyze the impact of network transmission delay on the migration target. Set the transmission delay threshold at 25ms to screen out storage nodes with shorter transmission delays. If the delay of storage node 1 is 22.4ms and the delay of 2 is 26ms, then only storage node 1 meets the network transmission requirements. Finally, determine storage node 1 as the migration target node for the write task of the data block, and obtain the migration target node for the write task of the data block.

[0045] The data block storage and archiving execution sub-module executes the storage and archiving operation of the data block based on the migration target node of the write task of the data block, calls the storage node to transfer and archive the write task, and completes the cloud multi-modal data archiving; Migrate the write task target node based on the data block and perform the storage and archiving operation of the data block. First, call the task management module of the storage system to mark the data block to be migrated as migratable on the original storage node. Then, send a write request to the target storage node 1 and verify whether the remaining storage space of the storage node is sufficient. Assume that the current available storage space of storage node 1 is 500G2, and the total size of the data block to be archived is 300G2. Then storage node 1 can accommodate this task. Next, start the data migration. The data block is transmitted through a network transport protocol (such as T3P / IP). During the transmission process, each data packet is attached with a checksum to ensure that the data is not damaged during transmission. For example, if the checksum of a certain data block is calculated as "0x13F5" before transmission, it should remain unchanged after being recalculated after being transmitted to the target node. If it is found that the checksum does not match, the transmission of this data block is requested again. After the data transmission is completed, the target storage node updates the storage directory and returns the write completion status. Finally, obtain the storage and archiving execution status.

[0046] The above is only the preferred embodiment of the present invention and does not limit the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, as long as it does not depart from the technical solution content of the present invention, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. Cloud-based Multimodal Data Dynamic Archiving System for Big Data, characterized in that, The system includes: The storage topology monitoring module obtains the connection status, bandwidth occupancy rate, and I / O load of storage nodes, calls the storage topology mapping table, compares the topology connection density of each storage node, calculates the topology weight of the storage node, and generates storage topology status data; The storage hierarchy dynamic adjustment module, based on the storage topology status data, counts the access times of multi-modal data within a set time window, calculates the access frequency threshold interval of the data, compares it with the storage hierarchy access priority reference value, analyzes the matching degree between the data storage hierarchy and the access frequency, determines the target position for data hierarchy adjustment, and generates a storage hierarchy adjustment plan; The data writing task decomposition module, based on the storage hierarchy adjustment plan, obtains the data block size of the data to be archived, combines the computing resources of the storage nodes, filters out the storage nodes with a computing resource occupancy rate lower than the set resource value, calculates the data block storage task allocation ratio according to the filtering results, and generates a writing task decomposition plan; The data consistency verification module applies the writing task decomposition plan, obtains the write confirmation status of the storage nodes, compares the data consistency between the storage nodes, calculates the data consistency deviation value, and generates a data consistency verification result.

2. The cloud-based multimodal data dynamic archiving system for big data according to claim 1, characterized in that, The storage topology status data includes the topology connection density of the storage node, the topology weight of the storage node, and the bandwidth utilization rate of the storage node. The storage hierarchy adjustment plan includes the data access frequency threshold interval, the storage hierarchy access priority reference value, and the target position for data hierarchy adjustment. The writing task decomposition plan includes the data block storage task allocation ratio, the filtering result of the storage node computing resource occupancy rate, and the target storage node list. The data consistency verification result includes the write confirmation status of the storage node, the data consistency deviation value between the storage nodes, and the data integrity verification result.

3. The cloud-based multimodal data dynamic archiving system for big data according to claim 1, characterized in that, The storage topology monitoring module includes: The connection status detection sub-module obtains the connection status data of the storage node, collects the connection information between the storage nodes, including the port status, connection validity, and transmission stability of the node, and judges the connection validity of each storage node based on the port connection success rate and connection delay parameters of the node, and calculates the connection stability data of each storage node; The bandwidth and I / O load analysis sub-module, based on the connection stability data of the storage node, collects the bandwidth occupancy and I / O load data of the storage node, obtains the current bandwidth usage rate, I / O request rate, and queue depth of the storage node, calculates the balance between the bandwidth occupancy and the I / O load, and obtains the bandwidth-I / O balance data of each storage node; The topology weight calculation sub-module, according to the bandwidth-I / O balance data, obtains the topology connection density of each storage node based on the topology mapping table of the storage topology, calculates the comprehensive influence value of the topology connection number of the storage node and its bandwidth-I / O balance degree, and uses the formula: ； Computing storage node The topological weight of , combined with the topological structure information of the storage node, generates storage topology state data, where represents the storage node and the node The number of topological connections between them represents the storage node The bandwidth-I / O balance degree value of represents the storage node The connection stability of represents the total number of storage nodes in the storage topology 4. The cloud-based multimodal data dynamic archiving system for big data according to claim 1, characterized in that, The storage hierarchy dynamic adjustment module includes: The access frequency statistics sub-module obtains the access times of multi-modal data within a set time window based on the stored topology state data, counts the access count values of each data block, calculates the access frequency of each data block, and calculates the access frequency change trend based on the time window to obtain the access frequency trend data of each data block; The storage level matching analysis sub-module calculates the access frequency threshold interval of the data based on the access frequency trend data, compares it with the access priority reference value of the storage level, determines whether the access frequency of each data block meets the access reference range corresponding to the storage level, calculates the matching degree between the data storage level and the access frequency, and obtains the storage matching degree data; The level adjustment target calculation sub-module determines the target position of the level adjustment of the data according to the storage matching degree data based on the matching degree between the data storage level and the access frequency, using the formula: ； Calculated data block 's adjusted target level value , combined with the storage topology state data, generate a storage level adjustment plan, where represents the access frequency trend value of the data block , represents the access priority reference value of the storage level , represents the current storage level stability of the data block , represents the total number of storage levels in the storage system.

5. The cloud-based multimodal data dynamic archiving system for big data according to claim 1, characterized in that, The data writing task decomposition module includes: The storage level data block acquisition sub-module determines the data archiving rules for different storage levels based on the storage level adjustment scheme, analyzes the current data occupancy of the storage device, extracts the data blocks that meet the archiving criteria, calculates the total amount and distribution ratio of the data blocks under each storage level according to the distribution of the data blocks, and obtains the data block distribution data of each storage level; The storage node computing resource screening sub-module screens all storage nodes based on the data block distribution data of each storage level, combines the computing resource occupancy information of the storage nodes, and extracts the storage nodes with a computing resource occupancy rate lower than the threshold according to the set threshold of the computing resource utilization rate to obtain the storage nodes available for task allocation; The storage task allocation ratio calculation sub-module calculates the available storage ratio of each storage node based on the storage nodes available for task allocation and the data block distribution data of each storage level, and combines the target data block size of the storage task, using the formula: ； Computing storage node The storage task allocation ratio , generate a write task decomposition plan, where represents the total amount of data blocks of the storage node in the storage hierarchy, represents the remaining computing resources of the storage node , represents the total remaining computing resources of available storage nodes, represents the total size of the storage task data blocks, represents the total number of storage nodes, represents the number of available storage nodes.

6. The cloud-based multimodal data dynamic archiving system for big data according to claim 1, characterized in that, The data consistency verification module includes: The storage node write confirmation sub-module applies the write task decomposition scheme, obtains the write confirmation status of all storage nodes, counts the number of successful writes and failed writes of each storage node, and calculates the write success rate of the storage node; The data consistency deviation calculation sub-module calculates the hash difference between the storage nodes by comparing the hash values of the data stored in different storage nodes based on the write success rate of the storage nodes, and combines the write timestamp difference of the storage blocks, using the formula: ； Calculate the data consistency deviation value , where represents the data hash value of the storage node , represents the data hash value of the reference storage node represents the data write timestamp of the storage node , represents the write timestamp of the reference storage node represents the total number of storage nodes; The consistency verification sub-module compares the data consistency deviation value with the set consistency threshold to determine whether the data meets the consistency standard and obtains the data consistency verification result.

7. The cloud-based multi-modal data dynamic archiving system for big data according to claim 1, wherein The system further includes an archiving task dynamic execution module; The archiving task dynamic execution module obtains the network transmission delay and load status data of the storage nodes based on the data consistency verification result, screens the storage nodes with a load status lower than the set load value, determines the target node for migrating the write task of the data block, and executes the storage archiving operation of the data block.

8. The cloud-based multi-modal data dynamic archiving system for big data according to claim 7, wherein The archiving task dynamic execution module includes: Based on the data consistency verification result, the storage node status monitoring sub-module obtains the network transmission delay and load status data of the storage node, calculates the real-time load occupancy rate of each storage node, calculates the average transmission delay of each storage node, and obtains the network transmission load status of the storage node. The write task migration target screening sub-module filters the storage nodes with a load occupancy rate lower than the set load value according to the network transmission load status of the storage node, calculates the load balance of each storage node, analyzes the influence degree of the transmission delay, and combines the available resource situation of the storage node to determine the storage node suitable as the write task migration target, and obtains the write task migration target node of the data block. Based on the write task migration target node of the data block, the data block storage and archiving execution sub-module executes the storage and archiving operation of the data block, calls the storage node to transfer and archive the write task, and completes the cloud multi-modal data archiving.

Images