A Distributed Storage and Management Method, System and Device for Ultra-Large-Scale Scientific and Technological Innovation Data

Abstract

The present invention relates to the technical field of data storage and management, and specifically, to a distributed storage and management method, system and device for ultra-large-scale scientific and technological innovation data. The method includes: obtaining ultra-large-scale scientific and technological innovation data based on a distributed storage architecture, splitting the data into multiple small pieces and distributing them to multiple storage nodes; performing compression and deduplication processing on the data in the storage nodes to generate an efficiently stored data sequence; based on the data sequence, automatically hierarchically storing the data using an intelligent algorithm; based on the result of the hierarchical storage, calculating the comprehensive load value of the nodes using a multi-factor collaborative optimization load balancing algorithm, and screening the data blocks in the high-load nodes for priority migration; combining the distributed storage architecture with an intelligent data management system to achieve data consistency management through a distributed consistency algorithm. The present invention enhances the reliability and scalability of the system and is applicable to scenarios such as scientific and technological research and development and big data analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage and management, and in particular, to a distributed storage and management method, system, and device for ultra-large-scale scientific and technological innovation data. Background Art

[0002] With the rapid development of scientific and technological innovation activities, the scale of various types of data generated in the field of scientific and technological innovation has grown exponentially, including experimental data, simulation data, multi-modal sensor data, and historical analysis data, etc. These data are characterized by complex structures, strong real-time performance, and large scales. However, traditional data storage and management systems face the following key technical problems when dealing with ultra-large-scale scientific and technological innovation data:

[0003] (1) Insufficient storage capacity. The growth rate of scientific and technological innovation data far exceeds the expansion ability of traditional storage systems. Existing systems are difficult to meet the requirements in terms of both capacity and performance, resulting in the gradual emergence of data storage bottlenecks, especially in high-concurrency access scenarios.

[0004] (2) High complexity in data management. The management of ultra-large-scale scientific and technological innovation data involves the efficient storage, dynamic scheduling, and real-time access of multi-dimensional and multi-modal data. Traditional centralized storage architectures are difficult to effectively address multiple requirements such as data consistency maintenance, distributed storage optimization, and data access efficiency.

[0005] (3) Poor system scalability. As the scale of scientific and technological innovation data grows, traditional storage systems are restricted by the architecture design during expansion, with insufficient horizontal expansion ability, unable to adapt to dynamic storage requirements and complex access patterns, resulting in a significant decline in storage performance and system reliability.

[0006] Currently, the industry is gradually exploring distributed storage architectures as an important means to meet the storage requirements of ultra-large-scale data. Through distributed node management, multi-node collaboration, and redundant storage mechanisms, the scalability and fault tolerance of storage systems are improved. At the same time, data compression and deduplication technologies can reduce the waste of storage space, and intelligent data management methods provide solutions for data hierarchical storage, load balancing, and fault recovery. However, the existing technologies still have the following deficiencies:

[0007] Lack of a dynamic hierarchical strategy for high-frequency access data and low-frequency access data, resulting in insufficient data access efficiency;

[0008] The algorithm performance of redundant data elimination and compression is insufficient and difficult to meet the large-scale processing requirements of scientific and technological innovation data;

[0009] The data consistency management method between distributed storage nodes has a high complexity, resulting in a reduction in system reliability and operating efficiency. Summary of the Invention

[0010] The present invention provides a method, system and device for distributed storage and management of ultra-large-scale scientific and technological innovation data, aiming to solve the problems of insufficient storage capacity, high management complexity and poor system scalability of ultra-large-scale scientific and technological innovation data through a distributed storage architecture, data compression and deduplication technologies, and intelligent data management, so as to optimize the storage space utilization rate, improve the data management efficiency and enhance the dynamic adaptability of the system.

[0011] To solve the above technical problems, the present invention provides a method for distributed storage and management of ultra-large-scale scientific and technological innovation data, including:

[0012] Based on a distributed storage architecture, obtain ultra-large-scale scientific and technological innovation data, split the data into multiple small pieces and distribute them to multiple storage nodes;

[0013] Compress and deduplicate the data in the storage nodes, identify redundant data and compress the valid data to generate an efficiently stored data sequence;

[0014] Based on the data sequence, use an intelligent algorithm to automatically store the data in layers, store the frequently accessed data in a high-speed storage medium, and store the infrequently accessed data in a low-cost storage medium;

[0015] Based on the results of the hierarchical storage, use a multi-factor collaborative optimization load balancing algorithm to calculate the comprehensive load value of the nodes, screen the data blocks in the high-load nodes for priority migration, and select target nodes to complete the data migration;

[0016] Based on the distributed storage architecture and the intelligent data management system, manage data consistency through a distributed consistency algorithm, and dynamically adjust the storage resource allocation according to real-time data requirements.

[0017] Further, the load balancing algorithm includes the following steps:

[0018] Collect the real-time status data of the storage nodes, including storage capacity utilization rate, bandwidth utilization rate, access frequency, and historical load fluctuation prediction value;

[0019] Use the comprehensive load calculation formula Generate a comprehensive load score for the nodes;

[0020] According to the node load score, screen the nodes with a load higher than the set threshold as the migration source nodes.

[0021] Further, the selection of the data migration target node includes the following steps:

[0022] Calculate the data block migration priority , and select the data blocks with a larger storage capacity and a lower access frequency as the migration objects;

[0023] Through the target node selection formula , filter out the target node with the minimum comprehensive migration cost;

[0024] Execute data migration and update the node status and data storage path index.

[0025] Furthermore, the steps of obtaining and distributing the storage of ultra-large-scale scientific and technological innovation data include:

[0026] Based on the distributed computing framework, split the ultra-large-scale scientific and technological innovation data according to data attributes or timestamps;

[0027] Dynamically allocate the data blocks to multiple storage nodes, and generate a data distribution strategy according to the capacity and load of the storage nodes.

[0028] Furthermore, the steps of redundant storage include:

[0029] Generate multiple redundant copies for each data block through the replication mechanism and disperse them for storage on different storage nodes;

[0030] Generate check information for each data block based on erasure coding technology and disperse it for storage on multiple storage nodes for data recovery.

[0031] Furthermore, the steps of data compression and deduplication processing include:

[0032] Use the feature hashing algorithm to detect redundant data in the data blocks;

[0033] Delete redundant data;

[0034] Compress the data blocks based on the content-aware compression algorithm to generate a compressed data sequence.

[0035] Furthermore, the steps of automatic hierarchical storage include:

[0036] Analyze the access frequency and importance of the data sequence to generate a high-priority data set and a low-priority data set;

[0037] Store the high-priority data in a high-speed storage medium and store the low-priority data in a low-cost storage medium;

[0038] Dynamically adjust the storage path and optimize the hierarchical structure of the storage medium.

[0039] Furthermore, the steps of hierarchical storage adjustment include:

[0040] Based on the data access logs, calculate the access frequency of the data blocks in real time;

[0041] Update the data priority according to the access frequency, and migrate the data block to the target storage medium.

[0042] Furthermore, a distributed storage and management system for ultra-large-scale scientific and technological innovation data includes:

[0043] A data acquisition and distribution module, configured to obtain ultra-large-scale scientific and technological innovation data from multiple data sources and distribute the data to multiple storage nodes;

[0044] A data compression and deduplication module, configured to perform compression and deduplication processing on the data in the storage nodes;

[0045] A hierarchical storage module, configured to hierarchically store data in a high-speed storage medium or a low-cost storage medium according to the access frequency and importance of the data;

[0046] A load balancing and fault recovery module, configured to dynamically adjust the data storage distribution and automatically recover data when a node fails;

[0047] A consistency management module, configured to achieve data consistency between multiple nodes through a distributed algorithm and dynamically adjust the storage resource configuration.

[0048] Furthermore, a distributed storage and management device for ultra-large-scale scientific and technological innovation data includes:

[0049] A distributed storage architecture, configured to obtain ultra-large-scale scientific and technological innovation data and distribute the data to multiple storage nodes;

[0050] A data processing unit, configured to perform the compression and deduplication operations on the data;

[0051] A hierarchical storage unit, configured to dynamically adjust the storage hierarchical structure based on the access frequency and importance of the data;

[0052] A system control unit, configured to monitor the running state of the storage nodes and manage data consistency through a distributed consistency algorithm;

[0053] A redundant storage unit, configured to generate redundant data copies or check information and distribute them for storage to multiple nodes.

[0054] The key innovation points of the present invention include:

[0055] (1) Distributed storage architecture design. A distributed storage system suitable for ultra-large-scale scientific and technological innovation data is constructed, combined with a data segmentation and redundant storage mechanism, supporting horizontal expansion and high availability.

[0056] (2) Data compression and deduplication technology. A redundant data identification method based on a feature hashing algorithm is proposed, and combined with content-aware compression technology, efficient compression and redundancy elimination of the stored data are achieved.

[0057] (3) Intelligent hierarchical storage algorithm. By analyzing the data access frequency and importance, the intelligent hierarchical storage algorithm is adopted to store high-frequency data in high-speed storage media, and at the same time, the hierarchical storage structure is dynamically adjusted to improve data access performance.

[0058] The following are its main beneficial effects:

[0059] (1) Improved storage efficiency. Through data compression and deduplication technologies, the present invention identifies and deletes redundant data in storage nodes, and efficiently compresses valid data, significantly reducing the storage space occupation. Compared with traditional storage methods, it optimizes the utilization rate of storage space and reduces storage costs at the same time.

[0060] (2) Optimized data access performance. By adopting the intelligent hierarchical storage technology, the storage structure is dynamically adjusted according to the data access frequency and importance. High-frequency access data is stored in high-speed storage media, and low-frequency access data is stored in low-cost storage media, effectively shortening the access latency of high-frequency data and improving the access performance of the overall storage system.

[0061] (3) Enhanced system reliability. Based on the distributed storage architecture, redundant storage of data is realized, and combined with the node monitoring and fault recovery mechanism, when a node fails, the data can be quickly located and migrated to a healthy node to ensure the high availability of data and the fault tolerance of the system. Description of the Drawings

[0062] Figure 1 It is a flowchart of a distributed storage and management method for ultra-large-scale scientific and technological innovation data provided by an embodiment of the present application;

[0063] Figure 2 It is a structural block diagram of a distributed storage and management system for ultra-large-scale scientific and technological innovation data provided by an embodiment of the present application;

[0064] Figure 3 It is a structural block diagram of a device provided by an embodiment of the present application. Specific Embodiments

[0065] Example 1: Refer to Figure 1 , which is a flowchart of a distributed storage and management method for ultra-large-scale scientific and technological innovation data provided by an embodiment of the present invention. This process can at least include steps S100 - S500:

[0066] S100. Based on the distributed storage architecture, obtain ultra-large-scale scientific and technological innovation data and perform distributed storage, allocate the data to multiple storage nodes, and achieve redundant storage of data to ensure high availability of data.

[0067] S200. Based on the data of the storage nodes, perform data compression and deduplication, identify redundant data and compress valid data, optimize the utilization rate of storage space, and generate an efficiently stored data sequence.

[0068] S300. Based on the data sequence after data compression and deduplication, use intelligent algorithms to automatically store the data in layers, store frequently accessed data in high-speed storage media, and store low-frequency data in media with lower costs.

[0069] S400. Based on the results of intelligent data hierarchical storage, perform load balancing and fault recovery, monitor the health status of storage nodes in real time, and dynamically adjust data storage and migration according to the load situation.

[0070] S500. Combine the distributed storage architecture with the intelligent data management system, optimize the storage performance and perform consistency management, achieve data consistency between multiple nodes through distributed algorithms, generate an optimal storage strategy, and adjust the storage resource allocation according to real-time data requirements.

[0071] Step S100 includes at least steps S110 - S130:

[0072] S110. Obtain ultra-large-scale scientific and technological innovation data, and divide the data into multiple small pieces through a distributed storage architecture and allocate them to multiple storage nodes.

[0073] First, obtain ultra-large-scale data from scientific and technological innovation data sources, including but not limited to multi-modal sensor data, experimental result data, etc. This data contains multiple dimensions of metrics, such as real-time working condition data such as temperature, pressure, stress, etc., and historical data.

[0074] Furthermore, based on the distributed storage architecture, use a distributed computing framework to preprocess and divide the ultra-large-scale scientific and technological innovation data. Specifically, the data is divided into multiple data blocks through specific algorithms (such as hash cutting method or timestamp cutting method), and the size of each data block is dynamically adjusted according to the node storage capacity.

[0075] Furthermore, allocate the divided data blocks to multiple storage nodes. The node allocation adopts a load balancing algorithm. Specifically, factors such as the storage capacity, load, and bandwidth of the nodes are used as input metrics for data allocation. It can be understood that the load status of node i is measured by the ratio of its current storage capacity to the bandwidth, and the calculation formula is as follows:

[0076]

[0077] Among them, is the load status of node i, is the current storage capacity of node i, is the bandwidth of node i.

[0078] S120. Redundantly store the data block. Through a replication mechanism or erasure coding technology, generate data redundancy replicas or parity information and perform backup storage on multiple nodes.

[0079] First, based on the storage allocation of the data block, use a replication mechanism or erasure coding technology (such as Reed - Solomon code) to redundantly store each data block. The replication mechanism generates multiple replicas of the data block and stores them on different nodes; while the erasure coding technology generates parity information so that even if some data is lost, the lost data can be recovered through other data blocks and parity information.

[0080] Specifically, assume the data block After being stored on node Generate a replica And store it on node . The generation ratio of the replica is determined according to the redundant storage requirement and the available storage capacity of the node. The number of replicas is related to the size of the data block, generally 2 to 3 replicas. The calculation method of the replica is:

[0081] ;

[0082] Among them, Represents the function for generating replicas, Is the original data block, Is the storage node, Is the replica data block.

[0083] Furthermore, when using erasure coding technology, the generated parity information Is calculated from multiple parts in the data block. Through the formula: ;

[0084] Among them, Represents the hash function of the erasure code, Is a component of the data block, and the generated parity information Is stored in different nodes for recovery when data is lost.

[0085] S130. Monitor the running status of the storage nodes, detect the health status of the storage nodes in real - time, generate a status report of the storage nodes, and dynamically adjust the data storage distribution according to factors such as node load and bandwidth to optimize the utilization of storage resources.

[0086] First, monitor the health status of each storage node in real - time, and obtain information such as the storage capacity, bandwidth usage, operating temperature, and CPU load of each node. Specifically, through a distributed monitoring system, the node status data will be uploaded to the central monitoring system in real - time.

[0087] Furthermore, based on the real-time monitoring data, the working status of each node is evaluated to generate a health assessment report for each node. This report uses the following evaluation formula according to parameters such as the node load and bandwidth:

[0088] ;

[0089] where, is the health assessment index of node i, is the node load, is the node bandwidth usage, is the node temperature, is the weight coefficient, representing the relative importance of each index.

[0090] Furthermore, based on the evaluation results, the resource allocation of the storage nodes is dynamically adjusted. When the health assessment index of a certain node exceeds the set threshold, the system will migrate part of the data of this node to other healthy nodes and reduce the load of this node. Specifically, the formula for dynamic adjustment is:

[0091]

[0092] where, is the data block to be migrated, is the healthy node.

[0093] Connection description: The data segmentation and allocation steps in S110 provide the basis for data distribution for subsequent redundant storage and monitoring adjustment. The segmentation and allocation information of each data block is used as input data in the redundant storage processing of S120.

[0094] S130 ensures the load balance of data blocks among storage nodes by real-time monitoring the node status, makes dynamic adjustments based on the health assessment report, thereby optimizing the data storage distribution and further ensuring the efficient utilization of storage resources.

[0095] Step S200 includes at least steps S210 - S230:

[0096] S210. Based on the data of the storage nodes, perform data compression and deduplication processing, identify and eliminate redundant data, and generate a compressed data sequence.

[0097] First, obtain the original data blocks after distributed storage allocation from the storage nodes , where i represents the storage node number and j represents the data block number. Furthermore, based on the node status report (the node load and bandwidth parameters generated by S130), preferentially process the node data with a health status greater than the threshold.

[0098] Further, for the said data block perform duplicate data detection and generate a unique identifier based on the feature hashing algorithm . Specifically, the calculation formula is:

[0099] ;

[0100] wherein, is the feature hash value of the data block, is the hash function used to identify redundant data.

[0101] Even further, compress the detected non-redundant data and generate a compressed data block using the content-aware compression algorithm . The compression ratio calculation formula is as follows:

[0102]

[0103] wherein, is the compression ratio, are the sizes of the data blocks before and after compression respectively. The generated will be used as the input data for subsequent processing.

[0104] S220. Perform deduplication on the said data block, identify the redundant parts in the data, delete the duplicate data, and generate a deduplicated data storage set.

[0105] First, based on the feature hash value generated in S210 , match the data block in the current storage node with the existing data. If the same hash value is found (where k, l are the numbers of other data blocks), then mark as a redundant data block.

[0106] Further, perform deletion processing on the marked redundant data block, retain only one copy, and store the reference record of the deleted redundant data into the index table . The index table format is:

[0107] ;

[0108] wherein, represents the storage node and location where the non-redundant data is located.

[0109] Further, define the set of data blocks retained after deduplication as , wherein:

[0110] The deduplicated data set is passed to S230 for further processing.

[0111] S230, based on the deduplicated and compressed data sequence, organizes it into an efficient data storage format to form an optimized stored data sequence.

[0112] First, according to the deduplicated data set , each data block is rearranged into an efficient storage format . Specifically, based on the status of the storage nodes (such as storage capacity and bandwidth utilization), the data blocks are reordered to optimize the read and write efficiency of data storage.

[0113] Furthermore, the data blocks are format-converted to generate a hierarchical storage index file . The index file contains the logical address and physical storage address of the data block, defined as follows:

[0114] ;

[0115] wherein, represents the address of the data block in the logical layer, represents the actual physical storage location.

[0116] Furthermore, finally, the stored data sequence in the efficient format and the index file are saved to the corresponding storage nodes, completing all steps of the data compression and deduplication module, providing optimized input for subsequent intelligent hierarchical storage (S300).

[0117] Step S300 includes at least steps S310 - S330:

[0118] S310, according to the access frequency and importance of the data, analyzes and identifies high-frequency data and low-frequency data from the compressed and deduplicated data sequence to generate data sets with different priorities.

[0119] First, obtain the storage address , logical address and access frequency of each data block from the optimized stored data sequence output by the S230 module and the index file . The calculation of the access frequency

[0120] is based on the log data of node real-time monitoring.Based on historical access log data and real-time request data, calculate the average access frequency of each data block using the following formula:

[0121]

[0122] Where, respectively represent the number of read requests and write requests for the data block , represents the total storage time of this data block.

[0123] Based on the access frequency and the importance index of the data (such as the source and use of the data block, etc.), generate a priority score :

[0124]

[0125] Where, and are weight factors. Divide the score of into a high-frequency data set H and a low-frequency data set L:

[0126] ;

[0127] θ is the priority threshold.

[0128] S320. Based on the analysis results of the high-frequency data and low-frequency data, use an intelligent algorithm to store the frequently accessed data in a high-speed storage medium and store the infrequently accessed data in a lower-cost storage medium.

[0129] First, after obtaining the high-frequency data set H and the low-frequency data set L, allocate storage resources according to the medium type of the storage node (the high-speed storage medium is a solid-state drive SSD, and the low-cost storage medium is a mechanical hard drive HDD). The target storage medium of each data block is determined according to the following rules: According to the following rules:

[0130] ;

[0131] Combined with the index file generated in S230 , adjust the physical storage path of each data block , migrate the high-frequency data H to the high-speed storage medium, and migrate the low-frequency data L to the low-cost storage medium.

[0132] Based on the optimized storage path , perform the data migration operation. The migration process will be based on the current load of the node and the health status Dynamically adjust the migration speed and order to avoid system overload.

[0133] S330. Dynamically adjust the data hierarchical storage structure according to the data access pattern and storage policy, and monitor the data access situation in real time.

[0134] First, use the monitoring system in S130 to continuously collect the real-time access situation of each data block, including the read / write request frequency , and the time distribution of requests .

[0135] Based on the change of the access pattern, re-evaluate the high-frequency data set and the low-frequency data set . Specifically, at regular intervals recalculate the priority score and update the data hierarchical storage set:

[0136] ;

[0137] For the data blocks with changed priorities , trigger the automatic migration operation, adjust its storage medium, and ensure that the hierarchical storage structure is consistent with the real-time access pattern.

[0138] Step S400 includes at least steps S410 - S430:

[0139] S410. Based on the multi-factor collaborative optimization model, by combining the real-time status of the node, the historical load trend, and the data priority, calculate the comprehensive load score of the node, and use the dynamic weight adjustment and prediction migration strategy to achieve an accurate load balancing report.

[0140] First, calculate the comprehensive load value of the node:

[0141]

[0142] Among them, is the comprehensive load score of node i; is the real-time storage capacity utilization rate of node i, is the maximum storage capacity of all nodes in the network; is the bandwidth utilization rate of node i, is the maximum bandwidth of all nodes in the network; is the access frequency of node i, is the average access frequency of all nodes in the network; is the predicted value of the historical load fluctuation of node i, predicted based on the long short-term memory network (LSTM); , , , is the dynamic weight coefficient, reflecting the influence weight of each factor on the comprehensive load.

[0143] Furthermore, calculate the priority of the data block:

[0144]

[0145] Among them, is the migration priority of data block j; is the storage size of data block j; is the access frequency of data block j; , is the priority weight.

[0146] Even further, select the data migration target node:

[0147]

[0148] Among them, is the minimum comprehensive migration cost of target node k; δ is the weight factor of bandwidth consumption during migration.

[0149] Specifically, the load balancing algorithm realizes the dynamic scheduling of storage nodes and data migration through a multi-factor collaborative optimization model. The specific steps are as follows:

[0150] ① Calculate the comprehensive load value of the node. Collect the real-time status data of all storage nodes, including storage capacity utilization rate, bandwidth utilization rate, access frequency, and historical load fluctuation prediction value. According to the comprehensive load calculation formula , generate a load score for each node.

[0151] ② Calculate the data migration priority. For nodes with high load, based on the data block priority calculation formula , screen out data blocks with larger storage capacity and lower access frequency as migration targets.

[0152] ③ Select the migration target node. Among the nodes with low load, use the comprehensive migration cost formula to select the optimal target node and complete the data migration operation.

[0153] ④ Dynamic adjustment and optimization. According to the real-time changes of the node status, dynamically adjust the weight coefficients , , , and , , to ensure the fast response ability of the load balancing strategy to sudden load fluctuations.

[0154] S420. Dynamically adjust the allocation and storage of data according to the load conditions of storage nodes, and automatically migrate data through a load balancing algorithm.

[0155] First, filter out overloaded nodes from the load status reports output by S410 and underloaded nodes . The overloading condition is:

[0156]

[0157] The underloading condition is:

[0158]

[0159] Among them, respectively represent the high threshold and low threshold of the load.

[0160] Furthermore, select the data set to be migrated from , and based on the size of the data and the importance index , preferentially migrate data with low importance and high capacity:

[0161]

[0162] Allocate to underloaded nodes , and allocate target nodes through the following rules :

[0163] ;

[0164] And update the storage path and the index file to complete the data migration.

[0165] <S430. When a node fails or the system is abnormal, based on the redundant storage and data backup mechanism, achieve automatic fault recovery and automatically migrate the data in the failed node to an available node.

[0166] First, use the monitoring system to check the health status of storage nodes in real time . If it is found that the health status of a certain node is lower than the fault threshold , mark as a failed node :

[0167] Furthermore, extract the set of copy data for redundant storage from , and for each data block ​​ During the recovery process, select the optimal recovery path according to the redundant storage strategy (replica mechanism or erasure code): If the replica mechanism is adopted, directly from the replica storage node Recovery:

[0168]

[0169] If erasure code is adopted, reconstruct the data through the parity information Reconstruct data:

[0170] )

[0171] Furthermore, migrate the recovered data to healthy nodes , and update the storage path and index file:

[0172] );

[0173] Step S500 includes at least steps S510 - S530:

[0174] S510. Based on the distributed storage architecture and data management strategy, generate a consistency management strategy through a distributed consensus algorithm to avoid data loss or conflict.

[0175] First, from the load status report output by S410 and the storage path update information of S430 , collect the real - time status of each node, including the current distribution of data replicas and the health status of the nodes . Furthermore, in combination with the redundant storage mechanism, extract the replica distribution information of each data block .

[0176] Furthermore, according to the stored data and replica distribution , verify the data consistency among multiple nodes. Specifically, detect consistency conflicts by comparing the hash values of the data on the master node and the replica data :

[0177]

[0178] If a conflict is detected, record the set of conflicting data blocks .

[0179] For the set of conflicting data blocks , generate a consistency repair strategy based on a distributed consensus algorithm (such as the Paxos or Raft protocol) The consistency repair strategy is executed according to the following rules:

[0180] If the data of the master node is complete, update the replica data: If the master node fails, select the data of a healthy node from the redundant replicas as the new master data: ;

[0181] S520. Combine the real-time data access situation and storage resources, adopt an intelligent scheduling mechanism to dynamically allocate storage resources, and optimize storage performance.

[0182] First, from the access frequency of S310 and the node load of S410 extract the real-time access situation of each data block . Specifically, through access log analysis, calculate the current access density of each data block :

[0183]

[0184] wherein, is the total storage time of this data block.

[0185] Furthermore, according to the access density , sort the data blocks by priority to generate a high-priority data set and a low-priority data set . The high-priority data set is allocated to high-performance storage nodes, and the low-priority data set is allocated to low-cost nodes. The selection of storage nodes is allocated according to the following rules:

[0186]

[0187] wherein, are the high and low thresholds of the access density respectively.

[0188] Furthermore, based on the above priority allocation results, migrate the data blocks to the target storage nodes and update the index file to record the new storage path .

[0189] S530. Automatically adjust the configuration of storage resources according to the consistency management strategy and storage resource allocation situation, optimize the expansion and allocation of storage according to real-time requirements, and generate an optimal storage strategy.

[0190] First, combine the consistency management strategy of S510 and the storage resource allocation result of S520 , dynamically adjust the resource configuration of storage nodes, including storage capacity Bandwidth allocation and computing power . The specific adjustment rules are as follows:

[0191] For high-load nodes, release some resources: ;

[0192] For low-load nodes, increase storage tasks:

[0193] Furthermore, if the current storage capacity is insufficient, trigger an expansion strategy to dynamically increase storage nodes. The expansion strategy is determined by detecting the utilization rate of the current storage nodes :

[0194] ;

[0195] When , add new nodes and allocate storage tasks.

[0196] Combine the expansion results and the consistency management strategy to generate an optimal storage strategy , and distribute this strategy to each storage node for implementation.

[0197] Example 2: Figure 2 Shows a structural block diagram of a distributed storage and management system for ultra-large-scale scientific and technological innovation data according to an embodiment of the present invention. As Figure 2 shown, the system may include:

[0198] The data acquisition and distribution module 10 is used to obtain ultra-large-scale scientific and technological innovation data from multiple data sources and perform preliminary segmentation and distribution of the data through a distributed storage architecture.

[0199] Collect data related to scientific and technological innovation research from data sources such as multimodal sensors, experimental platforms, and historical databases, including real-time working condition data (such as temperature, pressure, stress) and historical data.

[0200] Cut the collected raw data through a distributed storage architecture, divide the large data into several small pieces, and dynamically allocate them to different storage nodes using a load balancing strategy.

[0201] Implement data redundant storage, and back up data between multiple nodes through a replica mechanism or erasure code technology.

[0202] The data compression and deduplication module 20 performs compression and deduplication processing on the data in the distributed storage nodes to optimize the utilization rate of the storage space.

[0203] For the data blocks in the storage nodes, use a content-aware compression algorithm to reduce the storage size of the data while maintaining the integrity and accessibility of the data.

[0204] Through duplicate data detection technology, identify and eliminate redundant parts between data blocks, retain only one data copy, and generate corresponding index information.

[0205] Reorganize the compressed and deduplicated data blocks into an efficient storage format.

[0206] The intelligent hierarchical storage module 30 stores data in layers based on data access frequency and importance, and dynamically adjusts the allocation of storage media.

[0207] Extract data access patterns from the compressed and deduplicated data, and divide the data into high-priority data sets and low-priority data sets according to access frequency and importance.

[0208] Adopt an intelligent storage allocation algorithm to preferentially store frequently accessed data in high-speed storage media (such as SSDs), and store less frequently accessed data in lower-cost media (such as HDDs).

[0209] Monitor data access in real time and dynamically adjust the data layering strategy.

[0210] The load balancing and fault recovery module 40 monitors the load status of storage nodes, dynamically adjusts data distribution, and automatically recovers in case of node failures.

[0211] Based on the results of intelligent hierarchical storage, monitor the operating status of storage nodes in real time, including storage capacity, access frequency, computing power, etc., and generate a node load status report.

[0212] When the node load exceeds the threshold, migrate some data to low-load nodes based on the load balancing algorithm, and update the storage path and index information at the same time.

[0213] In case of a failure of a storage node, automatically recover the data through a redundant storage mechanism, and migrate the data of the failed node to a healthy node.

[0214] The optimization scheduling and consistency management module 50 combines a distributed storage architecture with an intelligent data management system to optimize system performance and achieve consistency management of data among multiple nodes.

[0215] Adopt a distributed consistency algorithm to verify the data in storage nodes, and avoid conflicts and inconsistencies between data copies.

[0216] Based on the real-time data access situation and the status of storage nodes, dynamically adjust the storage resource configuration to optimize the storage performance of the system.

[0217] Generate an optimal storage strategy, dynamically expand the storage capacity according to the system operation requirements, and reasonably allocate node storage tasks.

[0218] Embodiment 3: Figure 3 The structural block diagram of the device according to an embodiment of the present application is shown. As Figure 3 shown, the device includes:

[0219] The distributed storage architecture 41 acquires ultra-large-scale scientific and technological innovation data and distributes the data to multiple storage nodes.

[0220] The scientific and technological innovation data is sliced through the distributed computing framework, and data blocks are generated according to data attributes or timestamps.

[0221] The data blocks are dynamically allocated to multiple storage nodes, and a distribution strategy is generated according to the storage capacity and load status of the nodes.

[0222] The data processing unit 42 performs compression and deduplication operations on the data.

[0223] The redundant data is identified by using the feature hashing algorithm, and the duplicate content is deleted to reduce the storage space occupancy.

[0224] The data blocks are compressed based on the content-aware compression technology to generate an optimized data sequence.

[0225] The hierarchical storage unit 43 dynamically adjusts the storage hierarchical structure based on the access frequency and importance of the data.

[0226] The data access logs are analyzed, the priorities are divided according to the access frequency and importance of the data, and high-priority and low-priority data sets are generated.

[0227] The frequently accessed data is stored in a high-speed storage medium (such as an SSD), and the infrequently accessed data is stored in a low-cost storage medium (such as an HDD).

[0228] The storage hierarchical structure is dynamically adjusted to ensure storage efficiency and access performance.

[0229] The system control unit 44 monitors the running status of the storage nodes and manages data consistency through the distributed consistency algorithm.

[0230] The running status data of the nodes is collected in real time, including storage capacity, bandwidth utilization rate, and load information.

[0231] The data consistency between the storage nodes is verified based on the distributed consistency algorithm to avoid data conflicts or losses.

[0232] When a node fails, the failure is quickly located and the data is restored to a healthy node.

[0233] The redundant storage unit 45 generates data redundant copies or check information and dispersedly stores them in multiple nodes.

[0234] Generate check information based on erasure coding technology to reduce the space occupied by redundant storage.

Claims

1. A distributed storage and management method for ultra-large-scale scientific and technological innovation data, characterized in that, Including the following steps: Based on a distributed storage architecture, obtain ultra-large-scale scientific and technological innovation data containing real-time working condition data of multi-modal sensors, and divide the data into multiple small pieces by the hash cutting method or the timestamp cutting method, and allocate them to multiple storage nodes according to the load balancing algorithm; the expression of the load balancing algorithm is where is the load status of node i, is the current storage capacity of node i, is the bandwidth of node i; Adopt a replication mechanism or erasure coding technology to redundantly store data blocks in storage nodes. For the data in the storage nodes, generate a unique identifier using a feature hashing algorithm, compress the non-redundant data using a content-aware compression algorithm, and organize it into an efficient data storage format to form an optimized storage data sequence. The expression of the content-aware compression algorithm is: ； Among them, is the compression ratio, are the data block sizes before and after compression respectively; Based on the optimized stored data sequence, according to the access frequency and the importance index of the data generate a priority score ; The priority score includes a high-frequency data set and a low-frequency data set; Allocate storage resources according to the media type of the storage node, combine with a hierarchical storage index file, adjust the physical storage path of each data block, migrate high-frequency data to high-speed storage media, and migrate low-frequency data to low-cost storage media. The expression of the physical storage path is: ； Among them, is the target storage medium; SSD is a solid state drive; HDD is a hard disk drive; is a data block; H is a high-frequency data set; L is a low-frequency data set; Based on changes in the access pattern, every certain period of time recalculate the priority score and update the data hierarchical storage set; Based on a multi-factor collaborative optimization model, by combining the real-time state, historical load trend, and data priority of the nodes, calculate the comprehensive load score of the nodes, and use dynamic weight adjustment and predictive migration strategies to achieve an accurate load balancing report. The expression of the comprehensive load score is: ； Among them, is the comprehensive load score of node i; is the real-time storage capacity utilization rate of node i, is the maximum storage capacity of all nodes in the network; is the bandwidth utilization rate of node i, is the maximum bandwidth of all nodes in the network; is the access frequency of node i, is the average access frequency of all nodes in the network; is the predicted value of the historical load fluctuation of node i; is the dynamic weight coefficient; Based on a distributed storage architecture and data management strategy, generate a consistency management strategy through a distributed consensus algorithm, and optimize storage performance by combining real-time data access conditions and storage resources.

2. The method according to claim 1, characterized in that, The step of obtaining ultra-large-scale scientific and technological innovation data containing real-time working condition data of multi-modal sensors includes: Based on a distributed storage architecture, use a distributed computing framework to preprocess and split ultra-large-scale scientific and technological innovation data; Allocate the split data blocks to multiple storage nodes.

3. The method according to claim 1, wherein The step of redundant storage includes: The replication mechanism stores multiple copies of the data block on different nodes; The erasure coding technology generates check information so that even if some data is lost, the lost data can be restored through other data blocks and check information.

4. The method according to claim 1, characterized in that, The step of generating a priority score according to the access frequency and the importance index of the data further includes: Calculate the average access frequency of each data block according to historical access log data and real-time request data.

5. A distributed storage and management system for ultra-large-scale scientific and technological innovation data, which is applied to the distributed storage and management method for ultra-large-scale scientific and technological innovation data described in any one of the above claims 1-4, and is characterized in that, Including: A data acquisition and distribution module, which is used to obtain ultra-large-scale scientific and technological innovation data from multiple data sources and dynamically allocate it to different storage nodes using a load balancing strategy; A data compression and deduplication module, which is used to compress and deduplicate the data in the storage nodes; An intelligent hierarchical storage module, which hierarchically stores data based on data access frequency and importance, and dynamically adjusts the allocation of storage media; A load balancing and fault recovery module, which monitors the load status of storage nodes, dynamically adjusts data distribution, and automatically recovers in case of node failure; An optimization scheduling and consistency management module, which combines a distributed storage architecture and an intelligent data management system to optimize system performance and achieve data consistency management among multiple nodes.

6. A distributed storage and management device for ultra-large-scale scientific and technological innovation data, which is applied to the method for distributed storage and management of ultra-large-scale scientific and technological innovation data described in any one of the above claims 1-4, and is characterized in that, Including: A distributed storage architecture, which is used to obtain ultra-large-scale scientific and technological innovation data and allocate the data to multiple storage nodes; A data processing unit, which is used to perform the compression and deduplication operations of the data; A hierarchical storage unit, which dynamically adjusts the storage hierarchical structure based on the access frequency and importance of the data; A system control unit, which is used to monitor the running status of storage nodes and manage data consistency through a distributed consensus algorithm; A redundant storage unit, which is used to generate redundant data copies or check information and scatter them for storage on multiple nodes.

Images