Large-scale data efficient storage method, device and system

Abstract

The invention relates to the technical field of data storage, in particular to a large-scale data efficient storage method, device and system. The high-efficiency large-scale data storage method comprises the following steps: preprocessing to-be-stored data, and dividing the to-be-stored data into a plurality of small data blocks; according to the method, large-scale data are partitioned and stored in different types of storage media, so that the utilization rate of the storage space is remarkably improved, the storage cost is reduced, meanwhile, the problem of data consistency in a distributed storage environment is effectively solved, timely updating and high availability of the data are ensured, and the data storage efficiency is improved. The method and the device have the advantages that the method and the device are simple in structure, efficient data search and access can be realized by the aid of data indexing and load balancing algorithms, the system response speed can be increased, requirements of real-time data processing can be met, and the method and the device can flexibly adapt to different types of storage media and various data scenes and are good in expansibility and adaptability.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a method, device and system for efficiently storing large-scale data. Background Art

[0002] Data storage refers to the storage of data in various media or devices so that it can be accessed, retrieved, and used at any time in the future. Data storage involves writing information from computer memory or other storage media to physical media (such as hard drives, solid-state drives, tapes, optical disks, etc.) and reading data from them when needed. This ensures the long-term preservation and protection of data, as well as data processing, analysis, and recovery when needed. With the rapid development of information technology, the speed and scale of data generation are increasing, especially in the fields of cloud computing, big data, and the Internet of Things. The massive amount of data generated has made traditional data storage methods face more and more challenges.

[0003] However, traditional data storage solutions often cannot effectively address the following issues: 1. Low storage efficiency: Existing storage technologies often require a large amount of hardware resources when processing large-scale data, resulting in low storage efficiency and high costs; 2. Slow access speed: Fast data access is an important factor affecting the overall system performance, but traditional storage methods perform poorly in terms of reading and writing speeds and cannot meet the needs of real-time data processing; 3. Data redundancy and consistency issues: In a distributed storage environment, ensuring data consistency and managing redundant data is a major challenge, which can easily lead to untimely data updates or inconsistent storage.

[0004] Therefore, those skilled in the art provide a method, device and system for efficient storage of large-scale data to solve the problems raised in the above background technology. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device and system for efficient storage of large-scale data, so as to solve the problems of low storage efficiency, slow access speed, data redundancy and consistency mentioned in the above background technology.

[0006] In a first aspect, in order to solve the above technical problems, an embodiment of the present invention provides a method for efficiently storing large-scale data, comprising the following steps: S1, pre-processing the data to be stored and dividing it into multiple small data blocks; S2. storing the data blocks in different types of storage media according to the attributes and storage characteristics of the data blocks; S3, build data indexes to quickly find and access stored data blocks; S4. Use a dynamic load balancing algorithm to monitor the usage of storage media in real time and adjust the location of data blocks as needed.

[0007] In an optional implementation manner, the step S1 of preprocessing the data to be stored and dividing it into a plurality of small data blocks specifically comprises the following steps: S11. Data cleaning: remove redundant data and duplicate records, fill in missing values, obtain cleaned data, and ensure data quality and consistency; S12, data formatting: converting data into a unified format to obtain data in a unified format to ensure consistency and operability of subsequent processing; S13, data segmentation: set a threshold according to the data size and characteristics, and divide the data set into multiple small data blocks for easy storage and management. The characteristics include content characteristics and time characteristics. The content characteristics are segmented by subject or tag.

[0008] In an optional implementation manner, storing the data blocks in different types of storage media according to the attributes and storage characteristics of the data blocks in step S2 specifically includes the following steps: S21, attribute analysis: perform attribute analysis on each data block, including reading frequency, importance, and data type, and obtain analysis results to provide a basis for subsequent storage decisions; S22, storage medium selection: classifying the data blocks into different storage media according to the analysis results; S23. Data writing: Use appropriate API or protocol to store the data block in the corresponding medium and record the storage path.

[0009] In an optional implementation, the specific operation method for selecting the storage medium in step S2 is: storing hot data that is accessed frequently in a solid state drive, and storing cold data that is accessed infrequently in a mechanical hard drive.

[0010] In an optional implementation manner, the step S3 of establishing a data index to quickly search for and access the stored data blocks specifically includes the following steps: S31. Select index strategy: select a suitable data index structure to improve query efficiency; S32, index construction: Generate a unique ID for each data block, record related attributes and storage location, and build an index in the index structure; S33, index storage: the generated index is stored in memory or cache to speed up the search and updated regularly to facilitate fast retrieval; S34. Index update mechanism: The index mechanism is updated in a timely manner after each data storage or deletion to ensure the real-time and accuracy of the index.

[0011] In an optional implementation manner, the step S4 adopts a dynamic load balancing algorithm to monitor the usage of the storage medium in real time and adjust the position of the data block as needed, which specifically includes the following steps: S41, setting monitoring indicators: determining key monitoring indicators, the monitoring indicators including storage utilization, I / O performance and latency; S42, performing real-time data collection: using monitoring tools or custom scripts to regularly collect the monitoring indicators and store the data to form historical records; S43, load balancing calculation: using smooth flow control to analyze and determine the load of the storage medium; S44, data reallocation: determining whether to migrate a data block from one storage medium to another storage medium according to the load condition; S45, migration implementation: Migrate the data block to another storage medium. During the migration process, suspend access to the corresponding data block and update the index after completion at the new location.

[0012] In a second aspect, an embodiment of the present invention further provides a large-scale data efficient storage device, including: Data preprocessing unit: used to remove duplicates, fill missing values ​​and fix format errors in the data to be stored. It is also used to convert data from different sources and formats into a unified standard format. It is also used to divide the data to be processed into multiple small data blocks according to the size, type and content characteristics of the data; Storage management unit: used to analyze the characteristics of each data block, and intelligently select the most suitable storage medium for it according to the properties and characteristics of the data block. It is also responsible for safely writing the processed data block to the selected storage medium and recording the storage path and related metadata; Data index unit: used to select the appropriate index structure, generate a unique identifier for each data block, build detailed index data, and store it in memory or cache. It is also used to update the index information in real time after the data is stored or deleted; Load balancing unit: used to collect and monitor the key indicators of the usage rate, I / O performance and latency of each storage medium in real time, and generate monitoring reports. It is also used to implement intelligent load balancing algorithms. By analyzing monitoring data, it determines whether the storage location of data blocks needs to be adjusted. It is also used to automatically implement data block migration based on load balancing decisions, ensure that hot data is always stored on fast storage media, and update related index information. In an optional implementation manner, the data preprocessing unit internally integrates a data cleaning module, a data formatting module and a data segmentation module; The storage management unit is internally integrated with an attribute analysis module, a storage medium selection module and a data writing module; The data index unit internally integrates an index strategy selection module, an index construction module, an index storage module and an index update module; The load balancing unit internally integrates a monitoring indicator collection module, a load balancing calculation module, a data migration module and a scheduling management module.

[0013] In an optional implementation, the data cleaning module is used to remove redundant data and duplicate records and process missing values, the data formatting module is used to convert data from different sources into a unified format, and the data block module is used to divide the entire data set into multiple small data blocks according to set rules; The attribute analysis module is used to perform attribute analysis on each data block to evaluate its reading frequency, data importance and type; the storage medium selection module is used to automatically select a suitable storage medium for storage according to the attributes and storage characteristics of the data block; the data writing module is used to write the data block to the selected storage medium and record the relevant storage path and metadata; The index strategy selection module is used to select a suitable data index structure, the index construction module is used to generate a unique identifier for each data block and construct an index containing relevant attributes and storage locations, the index storage module is used to store the constructed index data in memory or cache, and the index update module is used to dynamically update the index when a data block is added, deleted or modified; The monitoring indicator collection module is used to collect the usage of storage resources in real time. The load balancing calculation module is used to implement a dynamic load balancing algorithm. By analyzing the monitoring data, it is determined whether the storage medium is overloaded and an adjustment plan is provided. The data migration module is used to implement the migration of data blocks according to the load balancing decision. The scheduling management module is used to schedule and coordinate the work of other modules to ensure efficient and smooth data processing, storage and migration. The other modules include a monitoring indicator collection module, a load balancing calculation module and a data migration module.

[0014] In a third aspect, an embodiment of the present invention further provides a large-scale data efficient storage system, including: A user interface module, which is used to provide interactive functions for storing and retrieving data; A memory, used for storing program instructions and for transmitting data between the user interface module; The processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the implementations of the first aspect according to the obtained program instructions.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1. In the present invention, large-scale data is divided into blocks, and the blocks are stored in different types of storage media according to the attributes and storage characteristics of the blocks. At the same time, data indexes are established to quickly find and access the stored data blocks. In addition, a dynamic load balancing algorithm is adopted to monitor the usage of the storage medium in real time and adjust the position of the data blocks as needed, thereby significantly improving the utilization of the storage space and reducing the storage cost. At the same time, the problem of data consistency in a distributed storage environment is effectively solved, ensuring the timely update and high availability of data.

[0017] 2. In the present invention, a data index module is used to construct an index of a data block, which facilitates quick search thereof. Meanwhile, a load balancing module is used to monitor the usage of storage resources and dynamically adjust them, thereby achieving efficient data search and access, improving the system response speed while meeting the needs of real-time data processing.

[0018] 3. In the present invention, by dividing the data into blocks and making it applicable to different storage media, the method and device can flexibly adapt to different types of storage media and various data scenarios, and have good scalability and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the steps of the method for efficient storage of large-scale data of the present invention.

[0020] Figure 2 This is a principle block diagram of the large-scale data efficient storage device of the present invention.

[0021] Figure 3 This is a principle block diagram of the large-scale data efficient system of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] like Figure 1As shown, an embodiment of the present invention provides a method for efficient storage of large-scale data. The method divides large-scale data into blocks and stores them in different types of storage media according to the attributes and storage characteristics of the data blocks. At the same time, a data index is established to quickly find and access the stored data blocks. In addition, a dynamic load balancing algorithm is adopted to monitor the usage of the storage medium in real time and adjust the position of the data blocks as needed, thereby significantly improving the utilization of the storage space and reducing the storage cost. At the same time, the problem of data consistency in a distributed storage environment is effectively solved, ensuring the timely update and high availability of data.

[0024] The specific steps include:

[0025] The first step is to preprocess the data to be stored and divide it into multiple small data blocks.

[0026] After receiving raw data from different sources (e.g., user comments from social media, sensor data, and transaction records), the system first preprocesses the data.

[0027] Perform data cleaning: Review the raw data to remove redundant user comments, fill in missing sensor data (such as missing temperature readings), and fix formatting errors (such as inconsistent date formats).

[0028] Data formatting: Convert all data into a unified format, such as unifying dates into the "YYYY-MM-DD" format to ensure consistency in subsequent processing.

[0029] Data chunking: After cleaning and formatting, the data is divided into multiple small data chunks. For example, multiple small data chunks of 100 records are formed, which can be stored and retrieved efficiently. For example, the first chunk contains 10 user comments, 40 sensor data, and 50 transaction records.

[0030] The second step is to store the data blocks in different types of storage media according to the attributes and storage characteristics of the data blocks. After the data blocks are divided, the system will perform attribute analysis to determine how to store the data blocks.

[0031] Perform attribute analysis: Assume that the first block of data contains multiple user comments. This type of data is frequently updated and accessed, so it should be stored in the SSD (solid state drive) to increase the read and write speed. The second block of data may contain historical sensor data, which is not frequently accessed and can be stored in the HDD (traditional mechanical hard disk).

[0032] Make storage decisions: Based on the analysis results, the system stores data blocks of user comments in SSDs and stores historical sensor data and transaction records in HDDs, ensuring efficient and cost-effective data storage.

[0033] The third step is to create a data index to quickly find and access the stored data blocks. After the data blocks are stored, the system will create an index for each data block.

[0034] Select the index structure: for example, a B-tree index may be created for the user comment data block to support fast range queries; for the sensor data block, a hash index may be used to quickly find temperature readings within a specific time period.

[0035] Index building: Generate a unique identifier for each data block and record its attributes (such as source, data type, storage path, etc.) in the index. For example, the index of user comments may contain information such as comment ID, comment content, and storage location.

[0036] Index storage: Index data is stored in memory to ensure fast response when querying.

[0037] The fourth step is to use a dynamic load balancing algorithm to monitor the usage of the storage medium in real time and adjust the location of the data block as needed. After the data block is successfully stored and indexed, the system enters the dynamic monitoring stage.

[0038] Real-time monitoring: The system regularly collects storage media usage information. For example, if the utilization rate of the SSD has reached 80%, while the utilization rate of the HDD is only 30%, the monitoring report will indicate that the SSD is facing an overload risk.

[0039] Make load balancing decisions: Using a dynamic load balancing algorithm, the system analyzes the current storage status and determines which infrequently accessed data blocks can be transferred, such as migrating some historical sensor data from the SSD to the HDD to free up SSD storage space.

[0040] Perform data migration: The system implements data migration to move specific sensor data blocks from the SSD to the HDD, and updates the corresponding index information to ensure that users can access them smoothly when querying.

[0041] Based on the same inventive concept, the embodiment of the present invention also provides a large-scale data efficient storage device, which can realize the functions corresponding to the aforementioned large-scale data efficient storage method. The large-scale data efficient storage device can be a hardware structure, a software module, or a hardware structure plus a software module. The large-scale data efficient storage device can be implemented by a chip system, which can be composed of chips or include chips and other discrete devices. See Figure 2, which is a structural diagram of a large-scale data efficient storage device provided in an embodiment of the invention, the large-scale data efficient storage device includes: a data preprocessing unit, a storage management unit, a data indexing unit and a load balancing unit.

[0042] The data preprocessing unit, storage management unit, data indexing unit and load balancing unit correspond to the specific operations for implementing the above steps one, two, three and four respectively.

[0043] The purpose of the data preprocessing unit is to improve the quality and consistency of the data to be stored. The data preprocessing unit can perform: 1. Data deduplication - When receiving user comment data from social media, the system first checks these comments and finds that some comments are repeated multiple times. After deduplication, the system only retains one copy of each unique comment. For example, the comment "This product is very good" by user "Zhang San" may appear 3 times in the data, and only one copy is retained after processing. 2. Fill missing values ​​- In the sensor data, temperature readings at certain time points are missing. The preprocessing unit fills these missing values ​​by using data from surrounding time points (such as interpolation) to ensure that complete data can be used for subsequent analysis. 3. Fix format errors - The date format in the original data may be different, some are "MM / DD / YYYY" and some are "YYYY-MM-DD". The preprocessing unit converts all dates to "YYYY-MM-DD" to ensure consistency. 4. Data standardization and division - The processed data is divided into multiple small data blocks according to type (such as comments, sensor data, transaction records). For example, 100 comments can be formed into one data block, and 50 sensor data can be formed into another data block, which is convenient for subsequent processing and storage.

[0044] The core of the storage management unit is to ensure the security and efficiency of data block storage. The storage management unit can perform: 1. Attribute analysis - after processing the data block, for example, the user comment data block is accessed frequently, while the historical sensor data is not frequently accessed. The storage management unit will analyze these characteristics and make the optimal storage decision for each data block. 2. Storage medium selection - the system will store frequently accessed user comment data blocks in the SSD to increase the reading speed; and store historical sensor data blocks in the HDD, because the access frequency of these data is relatively low, which can reduce storage costs. 3. Secure writing and metadata recording - after confirming the storage medium, the storage management unit will securely write the processed data blocks to the specified storage medium, and record the storage path and related metadata of each data block, such as storage time, block size and content type, to provide support for subsequent data query and management.

[0045] The goal of the data index unit is to quickly access and retrieve stored data. The data index unit is able to: 1. Index structure selection - the system selects B-tree or hash index based on the characteristics of the data block. In some cases, such as comment data blocks, the system selects B-tree structure to support range queries; while in sensor data, hash index is selected for fast search. 2. Unique identifier generation and index construction - each data block generates a unique ID, for example, "Comment Block_001" can be used as the identifier of the user comment data block, and its related attributes (such as storage location and data type) are recorded in the index. 3. Index storage and update - index data will be stored in memory for fast access. At the same time, any addition, deletion, or modification of data will trigger real-time update of the index. For example, if a comment is deleted, the system will immediately update the B-tree index to ensure data consistency.

[0046] The load balancing unit is responsible for ensuring efficient use of storage resources. The load balancing unit is able to: 1. Real-time monitoring - the system regularly monitors the utilization of SSDs and HDDs and finds that the utilization of SSDs has reached 85%, while the utilization of HDDs is only 40%. The load balancing unit will generate a monitoring report indicating that the SSD may be at risk of overload. 2. Intelligent load balancing algorithm - the system will analyze these monitoring data to determine whether the storage location of data blocks needs to be adjusted. If it is found that some less frequently accessed sensor data is still stored on the SSD, the system will decide to migrate it to the HDD to free up space on the SSD. 3. Automatic data migration - after confirming the migration of data blocks, the load balancing unit will automatically implement the migration, moving the historical sensor data blocks from the SSD to the HDD, and ensuring that the corresponding index information is updated to reflect the new storage location. This ensures that frequently accessed data remains on the fast storage media, while infrequently used data effectively frees up space.

[0047] The data preprocessing unit is internally integrated with a data cleaning module, a data formatting module, and a data segmentation module; the storage management unit is internally integrated with an attribute analysis module, a storage medium selection module, and a data writing module; the data indexing unit is internally integrated with an index strategy selection module, an index building module, an index storage module, and an index update module; the load balancing unit is internally integrated with a monitoring index collection module, a load balancing calculation module, a data migration module, and a scheduling management module. Each module is used to implement each operation step in the above method.

[0048] Based on the same inventive concept, the embodiment of the present invention also provides a large-scale data efficient storage system, see Figure 3, is a structural diagram of a large-scale data efficient storage system provided in an embodiment of the present invention, the large-scale data efficient storage system includes at least one user interface module, at least one processor, and a memory connected to the at least one processor, the user interface module and the memory maintain data communication through a data line, and the processor and the memory maintain data communication through a data line.

[0049] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for efficiently storing large-scale data, characterized in that: The following steps are involved: S1, pre-processing the data to be stored and dividing it into multiple small data blocks; S2. storing the data blocks in different types of storage media according to the attributes and storage characteristics of the data blocks; S3, build data indexes to quickly find and access stored data blocks; S4. Use a dynamic load balancing algorithm to monitor the usage of storage media in real time and adjust the location of data blocks as needed.

2. The method for efficient large-scale data storage according to claim 1, characterized in that: The step S1 of preprocessing the data to be stored and dividing it into a plurality of small data blocks specifically includes the following steps: S11. Data cleaning: remove redundant data and duplicate records, fill in missing values, and obtain cleaned data; S12, data formatting: converting the data into a unified format to obtain data in a unified format; S13, data segmentation: setting a threshold according to the data size and characteristics, and dividing the data set into multiple small data blocks, wherein the characteristics include content characteristics and time characteristics, and the content characteristics are segmented by subject or tag.

3. The method for efficient large-scale data storage according to claim 1, characterized in that: In step S2, storing the data blocks in different types of storage media according to the attributes and storage characteristics of the data blocks specifically includes the following steps: S21, attribute analysis: perform attribute analysis on each data block, including reading frequency, importance, and data type, and obtain analysis results; S22, storage medium selection: classifying the data blocks into different storage media according to the analysis results; S23. Data writing: Use appropriate API or protocol to store the data block in the corresponding medium and record the storage path.

4. The method for efficient large-scale data storage according to claim 3, characterized in that: The specific operation method for selecting the storage medium in step S2 is: storing the hot data with high frequency of access in the solid state hard disk, and storing the cold data with low frequency of access in the mechanical hard disk.

5. The method for efficient large-scale data storage according to claim 1, characterized in that: The step S3 of establishing a data index to quickly search and access the stored data blocks specifically includes the following steps: S31. Select index strategy: select a suitable data index structure; S32, index construction: Generate a unique ID for each data block, record related attributes and storage location, and build an index in the index structure; S33, index storage: store the generated index in memory or cache to speed up search and update it regularly; S34. Index update mechanism: The index mechanism is updated in a timely manner after each data storage or deletion.

6. The method for efficient large-scale data storage according to claim 1, characterized in that: The step S4 adopts a dynamic load balancing algorithm to monitor the usage of the storage medium in real time and adjust the position of the data block as needed. Specifically, the following steps are included: S41, setting monitoring indicators: determining key monitoring indicators, the monitoring indicators including storage utilization, I / O performance and latency; S42, performing real-time data collection: using monitoring tools or custom scripts to regularly collect the monitoring indicators and store the data to form historical records; S43, load balancing calculation: using smooth flow control to analyze and determine the load of the storage medium; S44, data reallocation: determining whether to migrate a data block from one storage medium to another storage medium according to the load condition; S45, migration implementation: Migrate the data block to another storage medium. During the migration process, suspend access to the corresponding data block and update the index after completion at the new location.

7. A large-scale data efficient storage device, wherein the large-scale data efficient storage device applies the large-scale data efficient storage method according to any one of claims 1 to 6, characterized in that: include: Data preprocessing unit: used to remove duplicates, fill missing values ​​and fix format errors in the data to be stored. It is also used to convert data from different sources and formats into a unified standard format. It is also used to divide the data to be processed into multiple small data blocks according to the size, type and content characteristics of the data; Storage management unit: used to analyze the characteristics of each data block, and intelligently select the most suitable storage medium for it according to the properties and characteristics of the data block. It is also responsible for safely writing the processed data block to the selected storage medium and recording the storage path and related metadata; Data index unit: used to select the appropriate index structure, generate a unique identifier for each data block, build detailed index data, and store it in memory or cache. It is also used to update the index information in real time after the data is stored or deleted; Load balancing unit: used to collect and monitor key indicators such as the utilization rate, I / O performance, and latency of each storage medium in real time, and generate monitoring reports. It is also used to implement intelligent load balancing algorithms and determine whether the storage location of data blocks needs to be adjusted by analyzing monitoring data. It is also used to automatically migrate data blocks based on load balancing decisions, ensure that hot data is always stored on fast storage media, and update related index information.

8. The large-scale data efficient storage device according to claim 7, characterized in that: The data preprocessing unit is internally integrated with a data cleaning module, a data formatting module and a data segmentation module; The storage management unit is internally integrated with an attribute analysis module, a storage medium selection module and a data writing module; The data index unit internally integrates an index strategy selection module, an index construction module, an index storage module and an index update module; The load balancing unit internally integrates a monitoring indicator collection module, a load balancing calculation module, a data migration module and a scheduling management module.

9. The large-scale data efficient storage device according to claim 8, characterized in that: The data cleaning module is used to remove redundant data and duplicate records and process missing values. The data formatting module is used to convert data from different sources into a unified format. The data block module is used to divide the entire data set into multiple small data blocks according to set rules. The attribute analysis module is used to perform attribute analysis on each data block to evaluate its reading frequency, data importance and type; the storage medium selection module is used to automatically select a suitable storage medium for storage according to the attributes and storage characteristics of the data block; the data writing module is used to write the data block to the selected storage medium and record the relevant storage path and metadata; The index strategy selection module is used to select a suitable data index structure, the index construction module is used to generate a unique identifier for each data block and construct an index containing relevant attributes and storage locations, the index storage module is used to store the constructed index data in memory or cache, and the index update module is used to dynamically update the index when a data block is added, deleted or modified; The monitoring indicator collection module is used to collect the usage of storage resources in real time. The load balancing calculation module is used to implement a dynamic load balancing algorithm. By analyzing the monitoring data, it is determined whether the storage medium is overloaded and an adjustment plan is provided. The data migration module is used to implement the migration of data blocks according to the load balancing decision. The scheduling management module is used to schedule and coordinate the work of other modules. The other modules include a monitoring indicator collection module, a load balancing calculation module and a data migration module.

10. A large-scale data efficient storage system, wherein the large-scale data efficient storage system applies the large-scale data efficient storage method according to any one of claims 1 to 6, characterized in that: include: A user interface module, which is used to provide interactive functions for storing and retrieving data; A memory, used for storing program instructions and for transmitting data between the user interface module; A processor is used to call the program instructions stored in the memory, and execute the steps included in the large-scale data efficient storage method described in any one of claims 1 to 5 according to the obtained program instructions.

Images