Cloud-based multimodal data dynamic archiving system for big data

Abstract

The present invention relates to the field of data archiving technology, including a cloud-based multimodal data dynamic archiving system for big data, wherein the system includes a storage topology monitoring module, a storage level dynamic adjustment module, a data writing task decomposition module, a data consistency verification module, and an archiving task dynamic execution module. In the present invention, by monitoring the status of storage nodes, optimizing the storage topology, improving the data storage balance and system throughput, dynamically adjusting the storage level, improving the storage resource utilization rate according to the data access frequency, adapting the computing resources of the storage nodes to perform data writing task allocation, avoiding storage hot spots, utilizing real-time data consistency verification, timely discovering and correcting storage anomalies, ensuring data reliability, considering the storage node network latency and load conditions for archiving tasks, optimizing data storage distribution, improving archiving efficiency and system stability, and jointly improving data processing efficiency to ensure high availability and scalability.

Description

Technical Field

[0001] The present invention relates to the field of data archiving technology, and in particular to a cloud-based multimodal data dynamic archiving system for big data. Background Art

[0002] The field of data archiving technology encompasses methods for long-term storage, management, and retrieval of digital information. Core content includes data collection, classification, storage, migration, and recovery to ensure the availability and integrity of data over long periods of time. Data archiving typically involves the conversion of different storage media, such as transferring frequently accessed data from high-speed storage devices to low-cost, long-term storage media. This area also encompasses methods such as data compression, deduplication, and index management to optimize storage space and improve retrieval efficiency. Furthermore, with the development of cloud computing and big data technologies, data archiving is gradually evolving toward a cloud-based distributed storage architecture, enabling data to be dynamically managed and accessed across multiple nodes, improving the flexibility and scalability of archiving systems.

[0003] Among them, the cloud-based multimodal data dynamic archiving system for big data refers to a method for dynamically managing and archiving different types of data in a cloud computing environment. It covers the classification and storage of multiple data types, including structured data, semi-structured data, and unstructured data, and uses data parsing and feature extraction methods to classify and archive data. During the data storage process, based on a distributed storage architecture, data is stored in blocks, and a cross-node storage strategy is used to achieve data load balancing and high availability. In addition, a scheduling mechanism based on timestamps and access frequency is used to achieve automatic data migration and lifecycle management, and a consistent hashing algorithm is combined to optimize data allocation strategies to improve the efficiency and reliability of data access.

[0004] In existing cloud-based multimodal data archiving processes, storage topology management lacks dynamic monitoring of storage node status, making it difficult to accurately assess storage node load. This can lead to overloads on some storage nodes and compromise data archiving stability. Storage tier adjustment relies on fixed policies and fails to optimize storage tiers as access frequency changes. This results in frequently accessed data being stored in low-priority tiers, impacting data retrieval performance. Storage task allocation fails to fully consider the computing resource usage of storage nodes, leading to task overloads on some storage nodes, impacting data write rates and increasing the overall burden on the storage system. Data consistency verification relies primarily on periodic offline testing, making it difficult to detect consistency issues during the storage process in real time. This can easily lead to undetected data corruption and compromise the integrity of archived data. During archiving tasks, the network transmission delay and load status of storage nodes are not fully considered, resulting in high-load nodes still being assigned storage tasks, reducing data archiving efficiency and increasing storage system instability. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the existing technology and propose a cloud-based multimodal data dynamic archiving system for big data.

[0006] To achieve the above objectives, the present invention adopts the following technical solutions: A cloud-based multimodal data dynamic archiving system for big data includes:

[0007] The storage topology monitoring module obtains the connection status, bandwidth usage, and I / O load of the storage nodes, calls the storage topology mapping table, compares the topological connection density of each storage node, calculates the topological weight of the storage node, and generates storage topology status data;

[0008] The storage hierarchy dynamic adjustment module counts the number of times multimodal data is accessed within a set time window based on the storage topology status data, calculates the data access frequency threshold interval, compares the storage hierarchy access priority benchmark value, analyzes the matching degree between the data storage hierarchy and the access frequency, determines the data hierarchy adjustment target location, and generates a storage hierarchy adjustment plan;

[0009] The data writing task decomposition module obtains the size of the data block to be archived based on the storage level adjustment plan, and screens the storage nodes whose computing resource occupancy rate is lower than the set resource value in combination with the storage node computing resources, calculates the data block storage task allocation ratio according to the screening results, and generates the writing task decomposition plan;

[0010] The data consistency verification module applies the write task decomposition scheme, obtains the write confirmation status of the storage node, compares the data consistency between the storage nodes, calculates the data consistency deviation value, and generates a data consistency verification result.

[0011] As a further solution of the present invention, the storage topology status data includes the topological connection density of the storage node, the topological weight of the storage node, and the bandwidth utilization of the storage node; the storage level adjustment scheme includes the data access frequency threshold interval, the storage level access priority baseline value, and the data level adjustment target position; the write task decomposition scheme includes the data block storage task allocation ratio, the storage node computing resource occupancy screening result, and the target storage node list; the data consistency verification result includes the storage node write confirmation status, the data consistency deviation value between storage nodes, and the data integrity verification result.

[0012] As a further solution of the present invention, the storage topology monitoring module includes:

[0013] The connection status detection submodule obtains the connection status data of the storage nodes and collects the connection information between the storage nodes, including the node port status, connection validity and transmission stability. Based on the node port connection success rate and connection delay parameters, it determines the connection validity of each storage node and calculates the connection stability data of each storage node.

[0014] The bandwidth and I / O load analysis submodule collects the bandwidth usage and I / O load data of the storage nodes based on the connection stability data of the storage nodes, obtains the current bandwidth utilization, I / O request rate and queue depth of the storage nodes, calculates the balance between bandwidth usage and I / O load, and obtains the bandwidth-I / O balance data of each storage node;

[0015] The topology weight calculation submodule obtains the topological connection density of each storage node based on the bandwidth-I / O balance data and the topology mapping table of the storage topology, and calculates the comprehensive impact value of the topological connection number of the storage node and its bandwidth-I / O balance using the formula:

[0016] ;

[0017] Compute and storage nodes Topological weight of , combined with the topological structure information of the storage node, the storage topology state data is generated, where, Represents a storage node With node The number of topological connections between Represents a storage node Bandwidth-I / O balance value, Represents a storage node The connection stability, Represents the total number of storage nodes in the storage topology.

[0018] As a further solution of the present invention, the storage level dynamic adjustment module includes:

[0019] The access frequency statistics submodule obtains the number of accesses to the multimodal data within a set time window based on the storage topology state data, counts the access count value of each data block, calculates the access frequency of each data block, and calculates the access frequency change trend according to the time window to obtain the access frequency trend data of each data block;

[0020] The storage hierarchy matching analysis submodule calculates the data access frequency threshold interval based on the access frequency trend data, compares it with the access priority reference value of the storage hierarchy, determines whether the access frequency of each data block meets the access reference range corresponding to the storage hierarchy, calculates the matching degree between the data storage hierarchy and the access frequency, and obtains storage matching degree data;

[0021] The level adjustment target calculation submodule determines the level adjustment target position of the data according to the storage matching data and the matching degree between the data storage level and the access frequency, using the formula:

[0022] ;

[0023] Calculation data block Adjust target level value , combined with the storage topology status data, a storage level adjustment plan is generated, where, Represents a data block The access frequency trend value of Represents the storage tier The access priority base value, Represents a data block The stability of the current storage tier, Represents the total number of storage tiers in the storage system.

[0024] As a further solution of the present invention, the data writing task decomposition module includes:

[0025] The storage-level data block acquisition submodule determines the data archiving rules for different storage levels based on the storage level adjustment plan, analyzes the current data occupancy of the storage device, extracts data blocks that meet the archiving standards, calculates the total amount and distribution ratio of data blocks at each storage level based on the distribution of data blocks, and obtains data block distribution data for each storage level;

[0026] The storage node computing resource screening submodule is based on the data block distribution data of each storage layer, combined with the computing resource occupancy information of the storage node, and sets a threshold according to the computing resource utilization rate. It screens all storage nodes, extracts storage nodes with computing resource occupancy rates below the threshold, and obtains storage nodes that can be used for task allocation;

[0027] The storage task allocation ratio calculation submodule calculates the available storage ratio of each storage node based on the storage nodes available for task allocation and the data block distribution data of each storage level, and uses the formula:

[0028] ;

[0029] Compute and storage nodes Storage task allocation ratio , generate a write task decomposition plan, where, Represents a storage node in the storage hierarchy The total amount of data blocks, Represents a storage node The remaining amount of computing resources, Represents the total remaining computing resources of available storage nodes, Represents the total storage task data block size, Represents the total number of storage nodes, Represents the number of available storage nodes.

[0030] As a further solution of the present invention, the data consistency verification module includes:

[0031] The storage node write confirmation submodule applies the write task decomposition scheme to obtain the write confirmation status of all storage nodes, count the number of successful writes and failed writes of each storage node, and calculate the write success rate of the storage node;

[0032] The data consistency deviation calculation submodule compares the data hash values stored in different storage nodes based on the write success rate of the storage node, calculates the hash difference between the storage nodes, and combines the write timestamp difference of the storage block to use the formula:

[0033] ;

[0034] Calculate data consistency deviation value ,in, Represents a storage node The data hash value, Represents the data hash value of the reference storage node, Represents a storage node The data is written with a timestamp, represents the write timestamp of the reference storage node, Represents the total number of storage nodes;

[0035] The consistency verification submodule compares the data consistency deviation value with the set consistency threshold value to determine whether the data meets the consistency standard and obtain the data consistency verification result.

[0036] As a further solution of the present invention, the system further includes an archiving task dynamic execution module;

[0037] The dynamic execution module of the archiving task obtains the network transmission delay and load status data of the storage node based on the data consistency verification result, filters the storage nodes whose load status is lower than the set load value, determines the target node for migrating the write task of the data block, and executes the storage archiving operation of the data block.

[0038] As a further solution of the present invention, the archiving task dynamic execution module includes:

[0039] The storage node status monitoring submodule obtains the network transmission delay and load status data of the storage node based on the data consistency verification result, calculates the real-time load occupancy rate of each storage node, calculates the average transmission delay of each storage node, and obtains the network transmission load status of the storage node;

[0040] The write task migration target screening submodule screens storage nodes whose load occupancy rate is lower than a set load value based on the network transmission load status of the storage nodes, calculates the load balancing status of each storage node, analyzes the impact of transmission delay, and determines the storage node suitable as the write task migration target based on the available resources of the storage node, and obtains the write task migration target node for the data block;

[0041] The data block storage archiving execution submodule migrates the target node based on the write task of the data block, executes the storage archiving operation of the data block, calls the storage node to transfer and archive the write task, and completes the cloud multimodal data archiving.

[0042] Compared with the prior art, the advantages and positive effects of the present invention are:

[0043] The present invention monitors the connection status, bandwidth occupancy, and I / O load of storage nodes to optimize the storage topology, improve the balance of data storage, reduce the congestion risk of high-load nodes, and enhance the throughput of the archiving system. The storage hierarchy is adjusted based on dynamic statistics of access frequency to ensure that high-frequency access data is stored in a storage layer with higher priority, and low-frequency data is migrated to more cost-effective storage media. This optimizes storage resource utilization and reduces unnecessary data migration overhead. Based on the computing resource occupancy of the storage nodes, suitable storage nodes are selected for data write task allocation, reducing storage hot spots and improving the scheduling efficiency of storage tasks. Data block storage tasks utilize computing resource screening and dynamic matching of storage task allocation ratios to balance storage task loads, improve data write efficiency, and reduce storage bottlenecks caused by storage node overload. During data consistency verification, the write confirmation status of the storage nodes and the calculation of data consistency deviations are combined to implement real-time data integrity detection, promptly detect storage anomalies, and ensure the reliability of archived data. During archiving task execution, target storage nodes are selected based on the network transmission delay and load of the storage nodes to prevent high-load nodes from affecting the execution efficiency of the archiving task. This optimizes the global distribution of data storage and enhances stability and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a system flow chart of the present invention;

[0045] Figure 2 The present invention stores the topology monitoring module flow chart;

[0046] Figure 3 This is a flow chart of the storage level dynamic adjustment module of the present invention;

[0047] Figure 4 This is a flowchart of the data writing task decomposition module of the present invention;

[0048] Figure 5 This is a flow chart of the data consistency verification module of the present invention;

[0049] Figure 6 This is a flow chart of the dynamic execution module of the archiving task of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] In the description of the present invention, it should be understood that the terms "length," "width," "up," "down," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inside," "outside," and the like, indicating positions or relationships, are based on the positions or relationships shown in the accompanying drawings and are intended only to facilitate the description of the present invention and simplify the description. They do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, in the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined.

[0052] See also Figure 1 The cloud-based multimodal data dynamic archiving system for big data includes:

[0053] The storage topology monitoring module obtains the connection status, bandwidth usage, and I / O load of the storage node, calls the topology mapping table of the storage topology, compares the topological connection density of each storage node, calculates the topological weight of the storage node, and generates storage topology status data;

[0054] The storage tier dynamic adjustment module counts the number of multimodal data accesses within a set time window based on storage topology status data, calculates the data access frequency threshold interval, compares the access priority baseline value of the storage tier, analyzes the match between the data storage tier and the access frequency, determines the target location for data tier adjustment based on the match, and generates a storage tier adjustment plan;

[0055] The data writing task decomposition module obtains the data block size of the data to be archived based on the storage tier adjustment plan. It then selects storage nodes with computing resource utilization rates lower than the set resource value based on the storage node computing resources. Based on the screening results, it calculates the storage task allocation ratio for the data block and generates a writing task decomposition plan.

[0056] The data consistency verification module applies the write task decomposition scheme, obtains the write confirmation status of the storage node, compares the data consistency between the storage nodes, calculates the data consistency deviation value, and generates the data consistency verification result;

[0057] The dynamic execution module of archiving tasks obtains the network transmission delay and load status data of the storage node based on the data consistency verification results, filters the storage nodes whose load status is lower than the set load value, determines the target node for migrating the data block write task, and executes the storage archiving operation of the data block.

[0058] The storage topology status data includes the topological connection density of the storage node, the topological weight of the storage node, and the bandwidth utilization of the storage node. The storage level adjustment plan includes the data access frequency threshold range, the storage level access priority baseline value, and the data level adjustment target location. The write task decomposition plan includes the data block storage task allocation ratio, the storage node computing resource occupancy screening results, and the target storage node list. The data consistency verification results include the storage node write confirmation status, the data consistency deviation value between storage nodes, and the data integrity verification results.

[0059] See also Figure 2 , the storage topology monitoring module includes:

[0060] The connection status detection submodule obtains the connection status data of the storage nodes and collects the connection information between the storage nodes, including the node port status, connection validity and transmission stability. Based on the node port connection success rate and connection delay parameters, it determines the connection validity of each storage node and calculates the connection stability data of each storage node.

[0061] Obtain the connection status data of the storage node. The connection information between storage nodes includes port status, connection validity, and transmission stability. In order to analyze the connection stability of the storage node, it is necessary to monitor the connection validity of the port and calculate its average number of valid connections and fluctuation standard deviation within a certain time interval. In the actual implementation process, the three ports of a storage node are sampled 10 times within the monitoring period of the past 60 seconds, and the number of valid connections of the ports is recorded in each sampling.

[0062] Table 1 Port connection validity record (unit: times)

[0063]

[0064] As shown in Table 1, the connection status of each port fluctuates slightly. In order to calculate the connection stability of the port, we first calculate the standard deviation of the connection stability of each port. .

[0065] Take port 1 as an example. Its mean is 54.7. Calculate its standard deviation:

[0066] ;

[0067] ;

[0068] Similarly, the standard deviation of ports 2 and 3 is calculated to obtain:

[0069] ;

[0070] Then, the connection stability is normalized and calculated:

[0071] ;

[0072] in, (Take the standard deviation of port P3) and bring it into the calculation:

[0073] ;

[0074] ;

[0075] ;

[0076] The results show that the connection stability of port 2 is the highest, while the connection fluctuation of port 3 is the largest and the connection status is the most unstable.

[0077] The bandwidth and I / O load analysis submodule collects the bandwidth usage and I / O load data of storage nodes based on the connection stability data of storage nodes, obtains the current bandwidth utilization, I / O request rate and queue depth of the storage nodes, calculates the balance between bandwidth usage and I / O load, and obtains the bandwidth-I / O balance data of each storage node.

[0078] The bandwidth and I / O load analysis submodule collects storage node bandwidth usage and I / O load data based on storage node connection stability data, obtains the storage node's current bandwidth utilization, I / O request rate, and queue depth, calculates the balance between bandwidth usage and I / O load, and obtains bandwidth-I / O balance data for each storage node;

[0079] Based on the storage node's connection stability data, we collect the storage node's bandwidth usage and I / O load data. We need to monitor each storage node's bandwidth usage and I / O rate, and calculate its fluctuations in different time periods to measure its balance. To more accurately analyze I / O load and bandwidth usage, we monitor the I / O rate and queue depth in different time periods. The data is as follows:

[0080] Table 2 I / O load monitoring data

[0081]

[0082] Bandwidth-I / O balance is calculated as follows:

[0083] ;

[0084] Substitute the data from Table 2 for calculation:

[0085] ;

[0086] ;

[0087] ;

[0088] The calculated Bandwidth-I / O Balance value is 20110, which represents the average balance of the storage node's I / O load over the five time periods. This indicates that the storage node received a large number of I / O requests during this period, and the relationship between its queue depth and request rate is relatively stable, indicating that the storage node's I / O resource utilization is relatively balanced. In actual applications, if the Bandwidth-I / O Balance value is too low, it may indicate underutilization of the storage device's I / O resources, affecting storage throughput. If the Bandwidth-I / O Balance value is too high, it may indicate that the storage device's I / O resources are overloaded, potentially affecting data read and write performance.

[0089] The topology weight calculation submodule obtains the topological connection density of each storage node based on the bandwidth-I / O balance data and the topology mapping table of the storage topology, and calculates the comprehensive impact value of the topological connection number of the storage node and its bandwidth-I / O balance using the formula:

[0090] ;

[0091] Compute and storage nodes Topological weight of , combined with the topological structure information of the storage node, the storage topology state data is generated, where, Represents a storage node With node The number of topological connections between Represents a storage node Bandwidth-I / O balance value, Represents a storage node The connection stability, Represents the total number of storage nodes in the storage topology;

[0092] According to the bandwidth-I / O balance data, based on the topology mapping table of the storage topology, the topological connection density of each storage node is obtained, and the comprehensive impact value of the topological connection number of the storage node and its bandwidth-I / O balance is calculated. Assume that the storage topology contains 5 storage nodes, and the connection density of each node is , Bandwidth-I / O balance and connection stability The monitoring values are as follows:

[0093] Table 3 Storage node topology parameters

[0094]

[0095] As shown in Table 3, the formula is used to calculate the storage nodes. Topological weight of:

[0096] ;

[0097] ;

[0098] This result shows that the storage node The topology weight is 7.18, and the stored topology status data can be used for subsequent optimization of the topology connection plan.

[0099] See also Figure 3 , the storage tier dynamic adjustment module includes:

[0100] The access frequency statistics submodule obtains the number of accesses to multimodal data within a set time window based on the storage topology status data, counts the access count value of each data block, calculates the access frequency of each data block, and calculates the access frequency change trend according to the time window to obtain the access frequency trend data of each data block;

[0101] Obtain the number of accesses to multimodal data within a set time window, count the access counts for each data block, calculate the access frequency of each data block, and calculate the access frequency trend based on the time window to obtain the access frequency trend value for each data block. For multimodal data, the set time window can be 1 hour, 1 day, or 1 week, depending on the application scenario, such as database storage, high-performance computing storage, or distributed storage systems. In a storage system, access count statistics can be completed through log analysis and I / O request logging. Assume that a storage node receives the following data access logs within 24 hours:

[0102] Table 4 Access log data

[0103]

[0104] As shown in Table 4, different data blocks have different access times in different time periods. To calculate the access frequency, assume that the access time window is 6 hours, and the calculation method is as follows:

[0105] Visit frequency = number of visits ÷ time window

[0106] For example, the access frequency of data block A1 = 150 ÷ 6 = 25 times / hour

[0107] Similarly calculate A2, A3, and A4:

[0108] A2: 230 ÷ 6 = 38.3 times / hour

[0109] A3: 180 ÷ 6 = 30 times / hour

[0110] A4: 200 ÷ 6 = 33.3 times / hour

[0111] When calculating the access frequency change trend, you can use the sliding window calculation method, such as calculating the change trend between A1 and A2:

[0112] ;

[0113] After calculating the trend values of other data blocks in sequence, the final access frequency trend data can be obtained.

[0114] The storage tier matching analysis submodule calculates the data access frequency threshold interval based on the access frequency trend data, compares it with the access priority benchmark value of the storage tier, determines whether the access frequency of each data block meets the access benchmark range corresponding to the storage tier, calculates the matching degree between the data storage tier and the access frequency, and obtains the storage matching degree data;

[0115] Based on the access frequency trend data, calculate the data access frequency threshold range, compare it with the access priority benchmark value of the storage layer, determine whether the access frequency of each data block meets the access benchmark range corresponding to the storage layer, calculate the matching degree between the data storage layer and the access frequency, and obtain the storage matching degree data. Assume that there are three layers in the storage system:

[0116] Tier 1: High-priority storage (SSD);

[0117] Tier 2: Medium-priority storage (HDD);

[0118] Tier 3: Low-priority storage (cold storage);

[0119] The access priority base values for each storage tier are defined as follows:

[0120] Level 1: ≥30 times / hour;

[0121] Level 2: 10-30 times / hour;

[0122] Level 3: ≤10 times / hour;

[0123] As shown in Table 5, the aforementioned access frequencies are compared with the hierarchical priority benchmark values to determine the matching degree.

[0124] Table 5 Storage matching calculation

[0125]

[0126] Formula for calculating matching degree:

[0127] ;

[0128] For example, the matching degree calculation of A3 (here Take the upper limit of the level as the calculation standard, that is, 30):

[0129] ;

[0130] A matching degree of 1 indicates that data block A3 matches the current level.

[0131] The hierarchical adjustment target calculation submodule determines the hierarchical adjustment target position of the data based on the storage matching data and the matching degree between the data storage hierarchy and the access frequency, using the formula:

[0132] ;

[0133] Calculation data block Adjust target level value , combined with the storage topology status data, a storage level adjustment plan is generated, where, Represents a data block The access frequency trend value of Represents the storage tier The access priority base value, Represents a data block The stability of the current storage tier, Represents the total number of storage tiers in the storage system;

[0134] Call the storage matching data, determine the target position for data level adjustment based on the matching degree between the data storage level and the access frequency, use the formula to calculate the adjustment target level value of each data block, and generate a storage level adjustment plan based on the storage topology status data.

[0135] For data block A2:

[0136] set up (Storage stability) = 0.9 (Storage Tier 1)

[0137] calculate:

[0138] ;

[0139] ;

[0140] ;

[0141] ;

[0142] Calculate the second part:

[0143] ;

[0144] ;

[0145] ;

[0146] ;

[0147] Compare storage tier adjustment targets:

[0148] like , the storage tier is adjusted to SSD;

[0149] like , the storage level is adjusted to HDD;

[0150] like , the storage level is adjusted to cold storage;

[0151] This calculation shows that data block A2 is suitable for the SSD storage tier. Therefore, the storage tier adjustment plan is as follows:

[0152] A1 holds the HDD;

[0153] A2 is migrated to SSD;

[0154] A3 maintains HDD;

[0155] A4 is migrated to SSD.

[0156] See also Figure 4 ,The data writing task decomposition module includes:

[0157] The storage-level data block acquisition submodule determines the data archiving rules for different storage levels based on the storage level adjustment plan, analyzes the current data occupancy of the storage devices, extracts data blocks that meet the archiving standards, calculates the total amount and distribution ratio of data blocks at each storage level based on the distribution of data blocks, and obtains data block distribution data for each storage level.

[0158] The hierarchical structure of a storage system typically includes different storage media, such as cache (e.g., SSD), disk storage (e.g., HDD), and cold storage (e.g., tape storage). Each medium has different access speeds and storage costs, so data archiving rules need to be tailored to the characteristics of the storage tiers. First, the current data occupancy of the storage device is retrieved to obtain the number of existing data blocks in each storage tier and determine their block size distribution. Using a hierarchical traversal approach, the storage system metadata is sequentially read to extract the data block distribution status for each storage tier. For example, assuming a storage system consists of two tiers, with the cache tier storing 20TB of data, the disk storage tier storing 80TB of data, and the cold storage tier storing 150TB of data, the data proportions of each storage tier are 11.8%, 47.1%, and 41.1%, respectively. When calculating the data block distribution ratio for each storage tier, we assume that in the 20TB of data stored in the cache tier, each data block is 4MB in size, resulting in 5,000,000 data blocks; in the disk storage tier, the data block size is 16MB, resulting in 5,000,000 data blocks; and in the cold storage tier, the data block size is 64MB, resulting in 2,343,750 data blocks. Based on this, the data block distribution ratio for each storage tier is calculated using the data block ratio, as shown in Table 6.

[0159] Table 6 Data block distribution ratio of each storage tier

[0160]

[0161] As shown in Table 6, the disk storage tier and cache tier have the same proportion of data blocks, but different data block sizes. The cold storage tier has relatively fewer data blocks. Combining the access frequency and data block access patterns of the storage tiers, we obtained data block distribution data for each storage tier.

[0162] The storage node computing resource screening submodule is based on the data block distribution data of each storage layer, combined with the computing resource occupancy information of the storage node, and sets a threshold according to the computing resource utilization rate. It then screens all storage nodes and extracts storage nodes with computing resource occupancy rates below the threshold, obtaining storage nodes that can be used for task allocation.

[0163] Based on the data block distribution ratios at each storage tier, the computing resource utilization information of the storage node is retrieved. The computing resource utilization of each storage node includes CPU utilization, memory utilization, and I / O bandwidth utilization. Assuming that the storage node has a total of 64 CPU cores, and the CPU utilization of each storage node at a given moment is 25%, 40%, 55%, and 70%, respectively, a computing resource utilization threshold of 50% is set. This threshold is based on the principle of load balancing management of computing resources. Specifically, when CPU utilization exceeds 50%, the execution efficiency of computing tasks is affected. Based on historical analysis of storage node operation status, when CPU utilization exceeds 50%, the average response time of I / O access requests increases by more than 1.5 times the original latency, and data write throughput decreases by more than 20%. Therefore, the setting of this threshold is related to the computing resource scheduling capabilities of the storage node and fluctuates with changes in the storage node's CPU frequency, memory bandwidth, and storage controller performance. Storage nodes with CPU utilization below this threshold are selected, namely, those with CPU utilizations of 25% and 40%.

[0164] Considering the I / O bandwidth utilization of storage nodes, assuming the total bandwidth of storage nodes is 10 Gbps, and the bandwidth utilization of each node is 3 Gbps, 5 Gbps, 6 Gbps, and 8 Gbps, respectively, a bandwidth utilization threshold of 60% is set. This threshold is determined based on the storage system's data transmission rate and the quality of service requirements for read and write requests. Typically, in storage system design, when bandwidth utilization exceeds 60%, the packet loss rate increases from 0.1% to over 1%, significantly degrading data access performance. This also reduces the ability to process parallel data streams and increases the queuing time for write tasks, thus affecting the overall efficiency of storage task execution. The adjustment range of this threshold is affected by the network bandwidth topology of the storage nodes, the packet scheduling policy, and the multi-channel data flow control capabilities within the storage nodes. Based on the set threshold, storage nodes with bandwidth utilization below 60% are selected, namely, those with bandwidth utilization of 3 Gbps and 5 Gbps. After taking the intersection, the storage nodes available for task allocation are determined, as shown in Table 7.

[0165] Table 7 Storage node screening table available for task allocation

[0166]

[0167] As shown in Table 7, storage nodes 1 and 2 with CPU usage below 50% and I / O bandwidth usage below 60% are finally selected to obtain the storage nodes that can be used for task allocation.

[0168] The storage task allocation ratio calculation submodule calculates the available storage ratio of each storage node based on the storage nodes available for task allocation and the data block distribution data of each storage layer. It also calculates the available storage ratio of each storage node based on the target data block size of the storage task and uses the formula:

[0169] ;

[0170] Compute and storage nodes Storage task allocation ratio , generate a write task decomposition plan, where, Represents a storage node in the storage hierarchy The total amount of data blocks, Represents a storage node The remaining amount of computing resources, Represents the total remaining computing resources of available storage nodes, Represents the total storage task data block size, Represents the total number of storage nodes, Represents the number of available storage nodes;

[0171] Based on the available storage nodes for task allocation and the data block distribution ratios of each storage tier, calculate the available storage ratio for each storage node. First, calculate the total number of data blocks on storage nodes 1 and 2. Assume that storage node 1 stores 1,500,000 data blocks and storage node 2 stores 1,200,000 data blocks, for a total of 2,700,000 data blocks. Based on the target data block size for the storage task, assume the total data block size for the task is 10 TB.

[0172] Assume that the remaining computing resources of storage node 1 are 20 CPU cores and the remaining computing resources of storage node B are 18 CPU cores. Then:

[0173] ;

[0174] ;

[0175] Calculate the storage task allocation ratio of storage node 1:

[0176] ;

[0177] ;

[0178] ;

[0179] Calculate the storage task allocation ratio of storage node 2:

[0180] ;

[0181] ;

[0182] ;

[0183] Finally, the storage task allocation ratio is 4.03 TB allocated to storage node 1 and 3.06 TB allocated to storage node 2. Then, the storage task allocation ratio of each storage node is obtained and a write task decomposition plan is generated.

[0184] See also Figure 5 , the data consistency verification module includes:

[0185] The storage node write confirmation submodule applies the write task decomposition scheme to obtain the write confirmation status of all storage nodes, count the number of successful writes and failed writes for each storage node, and calculate the write success rate of the storage node;

[0186] The write task decomposition scheme is invoked. First, the write confirmation status of all storage nodes is obtained. The storage node log information is extracted and analyzed. The timestamp, operation type, data identifier, and confirmation response code in the log records are analyzed. The write status of the storage nodes is classified and counted, distinguishing between successful and failed writes. The statistical data is aggregated for different storage nodes, and the write success rate of each storage node is calculated. Assuming a storage system contains five storage nodes, the total number of write requests and the number of successful writes for each storage node are shown in Table 8.

[0187] Table 8 Storage node write statistics

[0188]

[0189] As shown in Table 8, the write success rate is calculated as follows:

[0190] ;

[0191] Taking storage node 1 as an example, its write success rate is:

[0192] ;

[0193] Gets the write success rate of a storage node.

[0194] The data consistency deviation calculation submodule compares the data hash values stored in different storage nodes based on the write success rate of the storage node, calculates the hash difference between the storage nodes, and combines the write timestamp difference of the storage block with the formula:

[0195] ;

[0196] Calculate data consistency deviation value ,in, Represents a storage node The data hash value, Represents the data hash value of the reference storage node, Represents a storage node The data is written with a timestamp, represents the write timestamp of the reference storage node, Represents the total number of storage nodes;

[0197] Based on the write success rate of storage nodes, we compared the hash values of data stored on different storage nodes, extracted the hash values of the stored data on each storage node, and calculated the hash differences between storage nodes. This was then combined with the write timestamps of the storage blocks for quantitative analysis. Assume that three storage nodes store the same data block, but their hash values differ and their timestamps also differ to a certain extent, as shown in Table 9.

[0198] Table 9 Storage node data hash and timestamp table

[0199]

[0200] The consistency deviation value of the data is calculated using the formula in Table 9:

[0201] For storage node 1:

[0202] ;

[0203] For storage node 2:

[0204] ;

[0205] For storage node 3:

[0206] ;

[0207] Eventual consistency deviation value:

[0208] ;

[0209] Then the data consistency deviation value is obtained.

[0210] The consistency verification submodule compares the data consistency deviation value with the set consistency threshold to determine whether the data meets the consistency standard and obtain the data consistency verification result;

[0211] Based on the data consistency deviation value, it is compared with the set consistency threshold to determine whether the data meets the consistency standard. The data consistency threshold is set based on the data replication mechanism of the storage system, the data synchronization rate of the storage node, and the error tolerance of the storage medium. Usually, the storage system will set the consistency threshold based on the historical write success rate, hash matching rate, and write time deviation to ensure the synchronization stability of data between different storage nodes. In a high-reliability storage environment, the write success rate is usually maintained above 95%, the hash matching rate is close to 100%, and the write time deviation is generally controlled within 3 seconds. Therefore, the setting of the consistency threshold is usually adjusted according to the fluctuation range of these three parameters. In this example, the threshold is set to 5, that is, taking into account that the storage node may cause the hash deviation to accumulate to a certain extent due to factors such as network delay and I / O load imbalance, and 5 as a threshold can cover the common data storage consistency deviation range, while avoiding misjudgment caused by excessive strictness. If the calculated data consistency deviation value is less than or equal to 5, the data is judged to be consistent, otherwise the data is inconsistent. Combined with the data consistency deviation value calculated above , the value exceeds the set threshold, so the data is determined to be inconsistent. Get the data consistency verification result.

[0212] See also Figure 6 ,The archiving task dynamic execution module includes:

[0213] The storage node status monitoring submodule obtains the network transmission delay and load status data of the storage node based on the data consistency verification results, calculates the real-time load occupancy rate of each storage node, calculates the average transmission delay of each storage node, and obtains the network transmission load status of the storage node;

[0214] Based on the data consistency verification results, first obtain the network transmission delay and load status data of each storage node from the storage system. Specifically, it is necessary to call the network monitoring log of the storage node, extract the network transmission delay data of each storage node in different time periods, and collect the CPU occupancy, memory usage and disk I / O load status of the storage node to evaluate the current load of the storage node. In actual operation, the CPU usage of each node can be obtained through the SNMP protocol of the monitoring system (for example, the CPU occupancy of a storage node is 30%), and combined with the I / O throughput data (for example, the disk I / O throughput of a node is 150M2 / s) for calculation. After the real-time load occupancy of each storage node is counted, it needs to be normalized. To optimize the load of different resources, different weights are set for CPU usage, memory usage, and I / O throughput. The basis for weight setting is the impact of each resource on the performance of the storage task. CPU usage directly affects data processing capabilities, so the weight is set to 0.4. Memory usage affects data caching and write speed, so the weight is set to 0.3. I / O throughput determines the actual access speed of data, so the weight is set to 0.3. This setting value is determined by the architectural characteristics of the storage system. For example, for computing-intensive tasks, the CPU weight can be increased, while for large-scale data transmission tasks, the I / O throughput weight should be appropriately increased. Then, the comprehensive load usage is calculated. For example, for a storage node, its comprehensive load can be expressed as:

[0215] ;

[0216] In addition, the average network transmission delay of each storage node needs to be calculated. Assuming that the multiple measured delays of a storage node in the past minute are 20ms, 25ms, 22ms, 21ms, and 24ms respectively, its average network delay is calculated as follows:

[0217] ;

[0218] In this way, the network transmission load status of each storage node can be obtained, and the resource usage of the storage node can be evaluated to determine the subsequent task migration strategy, and finally the network transmission load status of the storage node can be obtained.

[0219] The write task migration target screening submodule selects storage nodes with load occupancy rates lower than the set load value based on the network transmission load status of the storage nodes, calculates the load balancing status of each storage node, analyzes the impact of transmission latency, and determines the storage node that is suitable as the write task migration target based on the available resources of the storage node, and obtains the write task migration target node for the data block;

[0220] The storage nodes are screened based on their network transmission load status. First, a load threshold is set to exclude storage nodes with a load status higher than the set value. The load threshold is set based on the storage node's processing capacity, historical average load level, and storage task scheduling strategy. Usually, the load rate of a storage node is mainly affected by 3PU occupancy, memory usage, and disk I / O throughput. When the load of a storage node exceeds a certain value, its task processing capacity will gradually decrease. To ensure the stability of data migration, the load threshold is usually set within the optimal range of the storage node's processing capacity. For example, in most enterprise-level storage systems, when the load rate exceeds 60%, the storage task response time will increase significantly, and when it is below 30%, the storage resources are not fully utilized. Therefore, 50% is taken as the load threshold to keep it in a state of balance between resource utilization and task scheduling. This value will be adjusted as the hardware specifications of the storage node change. For example, higher-performance storage nodes can accept higher load rates, while the load threshold of older storage nodes should be appropriately lowered. Assuming the load threshold is set to 50%, storage nodes with a load occupancy rate below 50% will be screened out. 0% of the storage nodes, for example, the load rate of storage node 1 is 38%, the load rate of storage node 2 is 47%, and the load rate of storage node 3 is 55%. Then only storage nodes 1 and 2 meet the migration target screening conditions. Among the filtered storage nodes, calculate the load balance of each storage node to avoid excessive concentration of task distribution. The load balance index can be evaluated by calculating the current task ratio of the storage node. Assume that the current number of storage tasks of storage node 1 is 100, accounting for 20%, and the current number of tasks of storage node 2 is 150, accounting for 3 0%, while the number of tasks on node 3 is 250, accounting for 50%. Since the load of node 3 has exceeded the threshold, only nodes 1 and 2 enter the migration target evaluation range. Then, the impact of network transmission delay on the migration target is analyzed, and the transmission delay threshold is set to 25ms to screen out storage nodes with shorter transmission delays. If the delay of storage node 1 is 22.4ms and the delay of 2 is 26ms, only storage node 1 meets the network transmission requirements. Finally, storage node 1 is determined as the migration target node for the data block write task, and the migration target node for the data block write task is obtained.

[0221] The data block storage and archiving execution submodule migrates the target node based on the data block write task, executes the data block storage and archiving operation, calls the storage node to transfer and archive the write task, and completes the cloud multimodal data archiving;

[0222] The target node is migrated based on the data block write task, and the data block storage archiving operation is performed. First, the storage system's task management module is called to mark the data block to be migrated from the original storage node as migratable. Then, a write request is issued to the target storage node 1, and the remaining storage space of the storage node is verified to be sufficient. Assuming that the current available storage space of storage node 1 is 500G2 and the total size of the data blocks to be archived is 300G2, storage node 1 can accommodate the task. Then, data migration begins, and the data blocks are transmitted via a network transmission protocol (such as T3P / IP). During the transmission process, each data packet is accompanied by a checksum to ensure that the data is not corrupted during transmission. For example, if the checksum of a data block is "0x13F5" before transmission, it should remain unchanged after recalculation after transmission to the target node. If the checksum does not match, the data block is retransmitted. After the data transmission is complete, the target storage node updates the storage directory and returns a write completion status, and finally obtains the storage archiving execution status.

[0223] The above are merely preferred embodiments of the present invention and do not limit the present invention in any other form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A cloud-based multimodal data dynamic archiving system for big data, characterized by: The system comprises: The storage topology monitoring module obtains the connection status, bandwidth usage, and I / O load of the storage nodes, calls the storage topology mapping table, compares the topological connection density of each storage node, calculates the topological weight of the storage node, and generates storage topology status data; The storage hierarchy dynamic adjustment module counts the number of times multimodal data is accessed within a set time window based on the storage topology status data, calculates the data access frequency threshold interval, compares the storage hierarchy access priority benchmark value, analyzes the matching degree between the data storage hierarchy and the access frequency, determines the data hierarchy adjustment target location, and generates a storage hierarchy adjustment plan; The data writing task decomposition module obtains the size of the data block to be archived based on the storage level adjustment plan, and screens the storage nodes whose computing resource occupancy rate is lower than the set resource value in combination with the storage node computing resources, calculates the data block storage task allocation ratio according to the screening results, and generates the writing task decomposition plan; The data consistency verification module applies the write task decomposition scheme, obtains the write confirmation status of the storage node, compares the data consistency between the storage nodes, calculates the data consistency deviation value, and generates a data consistency verification result; The storage level dynamic adjustment module includes: The access frequency statistics submodule obtains the number of accesses to the multimodal data within a set time window based on the storage topology state data, counts the access count value of each data block, calculates the access frequency of each data block, and calculates the access frequency change trend according to the time window to obtain the access frequency trend data of each data block; The storage hierarchy matching analysis submodule calculates the data access frequency threshold interval based on the access frequency trend data, compares it with the access priority reference value of the storage hierarchy, determines whether the access frequency of each data block meets the access reference range corresponding to the storage hierarchy, calculates the matching degree between the data storage hierarchy and the access frequency, and obtains storage matching degree data; The level adjustment target calculation submodule determines the level adjustment target position of the data according to the storage matching data and the matching degree between the data storage level and the access frequency, using the formula: ； Calculation data block Adjust target level value , combined with the storage topology status data, a storage level adjustment plan is generated, where, Represents a data block The access frequency trend value of Represents the storage tier The access priority base value, Represents a data block The stability of the current storage tier, Indicates the total number of storage tiers in the storage system.

2. The cloud-based multimodal data dynamic archiving system for big data according to claim 1 is characterized in that: The storage topology status data includes the topological connection density of the storage node, the topological weight of the storage node, and the bandwidth utilization of the storage node; the storage level adjustment plan includes the data access frequency threshold interval, the storage level access priority baseline value, and the data level adjustment target position; the write task decomposition plan includes the data block storage task allocation ratio, the storage node computing resource occupancy screening result, and the target storage node list; the data consistency verification result includes the storage node write confirmation status, the data consistency deviation value between storage nodes, and the data integrity verification result.

3. The cloud-based multimodal data dynamic archiving system for big data according to claim 1 is characterized in that: The storage topology monitoring module includes: The connection status detection submodule obtains the connection status data of the storage nodes and collects the connection information between the storage nodes, including the node port status, connection validity and transmission stability. Based on the node port connection success rate and connection delay parameters, it determines the connection validity of each storage node and calculates the connection stability data of each storage node. The bandwidth and I / O load analysis submodule collects the bandwidth usage and I / O load data of the storage nodes based on the connection stability data of the storage nodes, obtains the current bandwidth utilization, I / O request rate and queue depth of the storage nodes, calculates the balance between bandwidth usage and I / O load, and obtains the bandwidth-I / O balance data of each storage node; The topology weight calculation submodule obtains the topological connection density of each storage node based on the bandwidth-I / O balance data and the topology mapping table of the storage topology, and calculates the comprehensive impact value of the topological connection number of the storage node and its bandwidth-I / O balance using the formula: ； Compute and storage nodes Topological weight of , combined with the topological structure information of the storage node, the storage topology state data is generated, where, Represents a storage node With node The number of topological connections between Represents a storage node Bandwidth-I / O balance value, Represents a storage node The connection stability, Represents the total number of storage nodes in the storage topology.

4. The cloud-based multimodal data dynamic archiving system for big data according to claim 1 is characterized in that: The data writing task decomposition module includes: The storage-level data block acquisition submodule determines the data archiving rules for different storage levels based on the storage level adjustment plan, analyzes the current data occupancy of the storage device, extracts data blocks that meet the archiving standards, calculates the total amount and distribution ratio of data blocks at each storage level based on the distribution of data blocks, and obtains data block distribution data for each storage level; The storage node computing resource screening submodule is based on the data block distribution data of each storage layer, combined with the computing resource occupancy information of the storage node, and sets a threshold according to the computing resource utilization rate. It screens all storage nodes, extracts storage nodes with computing resource occupancy rates below the threshold, and obtains storage nodes that can be used for task allocation; The storage task allocation ratio calculation submodule calculates the available storage ratio of each storage node based on the storage nodes available for task allocation and the data block distribution data of each storage level, and uses the formula: ； Compute and storage nodes Storage task allocation ratio , generate a write task decomposition plan, where, Represents a storage node in the storage hierarchy The total amount of data blocks, Represents a storage node The remaining amount of computing resources, Represents the total remaining computing resources of available storage nodes, Represents the total storage task data block size, Represents the total number of storage nodes, Indicates the number of available storage nodes.

5. The cloud-based multimodal data dynamic archiving system for big data according to claim 1 is characterized in that: The data consistency verification module includes: The storage node write confirmation submodule applies the write task decomposition scheme to obtain the write confirmation status of all storage nodes, count the number of successful writes and failed writes of each storage node, and calculate the write success rate of the storage node; The data consistency deviation calculation submodule compares the data hash values stored in different storage nodes based on the write success rate of the storage node, calculates the hash difference between the storage nodes, and combines the write timestamp difference of the storage block to use the formula: ； Calculate data consistency deviation value ,in, Represents a storage node The data hash value, Represents the data hash value of the reference storage node, Represents a storage node The data is written with a timestamp, represents the write timestamp of the reference storage node, Represents the total number of storage nodes; The consistency verification submodule compares the data consistency deviation value with the set consistency threshold value to determine whether the data meets the consistency standard and obtain the data consistency verification result.

6. The cloud-based multimodal data dynamic archiving system for big data according to claim 1 is characterized in that: The system also includes an archiving task dynamic execution module; The dynamic execution module of the archiving task obtains the network transmission delay and load status data of the storage node based on the data consistency verification result, filters the storage nodes whose load status is lower than the set load value, determines the target node for migrating the write task of the data block, and executes the storage archiving operation of the data block.

7. The cloud-based multimodal data dynamic archiving system for big data according to claim 6 is characterized in that: The archiving task dynamic execution module includes: The storage node status monitoring submodule obtains the network transmission delay and load status data of the storage node based on the data consistency verification result, calculates the real-time load occupancy rate of each storage node, calculates the average transmission delay of each storage node, and obtains the network transmission load status of the storage node; The write task migration target screening submodule screens storage nodes whose load occupancy rate is lower than a set load value based on the network transmission load status of the storage nodes, calculates the load balancing status of each storage node, analyzes the impact of transmission delay, and determines the storage node suitable as the write task migration target based on the available resources of the storage node, and obtains the write task migration target node for the data block; The data block storage archiving execution submodule migrates the target node based on the write task of the data block, executes the storage archiving operation of the data block, calls the storage node to transfer and archive the write task, and completes the cloud multimodal data archiving.

Images