Intelligent data storage management system

Abstract

The invention discloses an intelligent data storage management system which comprises an intelligent data management platform, the intelligent data management platform comprises a pooling storage identification module, a data classification module, a data migration module and a flow-saving analysis module, and the invention relates to the technical field of data management. According to the intelligent data storage management system, hot data, cold data and cloud backup are matched, the data are stored in a proper position according to the access frequency and the use requirement of the data, the data storage efficiency and the access speed are improved, meanwhile, the safety and the reliability of the data are ensured, and through access frequency monitoring, frequency sorting and dynamic standard drawing, the data storage efficiency is improved. According to the method, classification can be automatically carried out according to the access frequency of the data, the accuracy and reliability of the data are ensured while the working efficiency is improved, powerful support is provided for intelligent migration of the data, and intelligent management of data storage is achieved while the method has the advantages of being high in storage efficiency, high in access speed, intelligent in backup and intelligent in migration.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to an intelligent data storage management system. Background Art

[0002] Data storage refers to the temporary files generated during the processing of data streams or the information to be searched during the processing. The commonly used storage media are disks and tapes. The storage organization method varies depending on the storage media. On tapes, data can only be accessed in sequential file mode; on disks, sequential access or direct access can be used according to usage requirements. The data storage method is closely related to the data file organization. The key lies in establishing the correspondence between the logical and physical orders of records and determining the storage address to improve the data access speed.

[0003] Traditional data storage methods often use solid-state drives or mechanical hard drives for data storage. Their storage methods are fixed, resulting in problems such as low storage efficiency, slow data access speed in traditional data storage methods. Moreover, data backups are also stored in solid-state drives or mechanical hard drives. Once hardware damage occurs, data restoration is often impossible. Therefore, an intelligent data storage management system is specifically proposed. It divides data into hot data or not according to the access frequency of the data, and at the same time combines the data volume of hot data and the size of the hot data storage space capacity to dynamically screen hot data, ensuring the data access speed. At the same time, using cloud backup methods to ensure data security and providing security guarantees for the migration of hot data and cold data, achieving the purpose of improving data storage efficiency, accelerating data access speed, and simplifying the data backup and migration process, and realizing the intelligent management of data storage. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the present invention provides an intelligent data storage management system, which solves the problems of low storage efficiency, slow data access speed, and easy data damage in traditional data storage methods.

[0005] To achieve the above objectives, the present invention is realized through the following technical solutions: An intelligent data storage management system includes an intelligent data management platform. The intelligent data management platform includes a pooled storage identification module, a data classification module, a data migration module, and a traffic-saving analysis module. The pooled storage identification module is respectively docked with the data classification module and the traffic-saving analysis module. The data classification module is docked with the data migration module. The data migration module is docked with the pooled storage identification module. The data migration module is docked with the traffic-saving analysis module.

[0006] The present invention is further configured such that: the pooling storage identification module includes a hot data storage unit, a cloud backup unit, and a cold data storage unit. The hot data storage unit is connected to the cloud backup unit, the cloud backup unit is connected to the cold data storage unit, and the cold data storage unit is connected to the hot data storage unit.

[0007] The present invention is further configured such that: the hot data storage unit is used to store hot data, that is, data that needs to be accessed quickly;

[0008] The cold data storage unit is used to store cold data, that is, data that does not need to be accessed quickly;

[0009] The cloud backup unit is used to perform cloud backup on the data stored in the hot data storage unit and the cold data storage unit, and perform regular updates.

[0010] The present invention is further configured such that: the data classification module includes an access frequency monitoring unit, a frequency sorting unit, and a dynamic standard formulation unit. The access frequency monitoring unit is connected to the frequency sorting unit, and the frequency sorting unit is connected to the dynamic standard formulation unit.

[0011] The present invention is further configured such that: the access frequency monitoring unit is used to calculate the access frequency of different data within a unit time according to the operation log;

[0012] The frequency sorting unit is used to sort a number of data in descending order of data access frequency, and obtain the corresponding data volume size;

[0013] The dynamic standard formulation unit is used to calculate the sum of the data volume sizes corresponding to a number of consecutive data ranked at the front, and screen out a number of data combinations that meet the capacity of the hot data storage unit as the hot data combination {a1, a2, …, a n}.

[0014] The present invention is further configured such that: the method for screening a number of data combinations that meet the capacity of the hot data storage unit includes:

[0015] MAX(A1 + A2 + … + A n ) ≤ T

[0016] wherein, A1 is the data volume size corresponding to the data a1 with the highest access frequency, A n is the data volume size corresponding to the data a n with the nth highest access frequency, and T is the capacity of the hot data storage unit.

[0017] The present invention is further configured such that: the data migration module includes a data classification identification unit, an allocation logic authentication unit, and a backup migration unit. The data classification identification unit is docked with the allocation logic authentication unit, and the allocation logic authentication unit is docked with the backup migration unit.

[0018] The present invention is further configured such that: the data classification identification unit is used to perform hot data identification on all data in the hot data combination, locate the current storage location of the data with hot data identification, perform cold migration identification on the data without hot data identification stored in the hot data storage unit, and perform hot migration identification on the data with hot data identification stored in the cold data storage unit;

[0019] The allocation logic authentication unit is used to upload the data with cold migration identification and hot migration identification to the cloud backup unit, perform backup update on the corresponding backup data of the data, and delete the data with cold migration identification in the hot data storage unit and the data with hot migration identification in the cold data storage unit;

[0020] The backup migration unit is used to migrate the data with hot migration identification in the cold data storage unit to the hot data storage unit, and then download the backup data corresponding to the data with cold migration identification from the cloud backup unit to the cold data storage unit.

[0021] The present invention is further configured such that: the traffic-saving analysis module includes a migration frequency analysis unit and a fixed update unit. The migration frequency analysis unit is docked with the fixed update unit.

[0022] The present invention is further configured such that: the migration frequency analysis unit is used to screen the data with hot migration identification that is the same as the data with cold migration identification within adjacent unit time. When the same situation occurs continuously for a set number of times, a resident hot data identification is added to the data, and a secondary screening is performed on the method of screening several data combinations that meet the capacity of the hot data storage unit. The secondary screening method includes:

[0023] MAX(A1+A2+…+A n )≤T-A G

[0024] where A G is the data volume size corresponding to the data with resident hot data identification;

[0025] The fixed update unit is used to cancel the resident hot data identification of the data when the data with resident hot data identification does not exist in the hot data combination {a1, a2, …, a n} continuously for a set number of times.

[0026] The present invention provides an intelligent data storage management system. It has the following beneficial effects:

[0027] (1) By means of the coordinated settings of hot data, cold data, and cloud backup, the present invention stores data in appropriate locations according to the access frequency and usage requirements of the data, improving the data storage efficiency and access speed. At the same time, it ensures the security and reliability of the data. Through access frequency monitoring, frequency sorting, and dynamic standard formulation, it can automatically classify data according to the access frequency, improving work efficiency while ensuring the accuracy and reliability of the data, providing strong support for the intelligent migration of data, and achieving intelligent management of data storage while having the advantages of high storage efficiency, fast access speed, intelligent backup, and intelligent migration.

[0028] (2) The present invention further optimizes the data migration process through the traffic-saving analysis module. By identifying and processing frequently migrated data, unnecessary migration operations are reduced, thereby saving network traffic and storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of the system architecture of the present invention;

[0030] Figure 2 is a schematic diagram of the system architecture of the pooled storage identification module of the present invention;

[0031] Figure 3 is a schematic diagram of the system architecture of the data classification module of the present invention;

[0032] Figure 4 is a schematic diagram of the system architecture of the data migration module of the present invention;

[0033] Figure 5 is a schematic diagram of the system architecture of the traffic-saving analysis module of the present invention.

[0034] In the figure:

[0035] 1. Intelligent data management platform;

[0036] 2. Pooled storage identification module; 201. Hot data storage unit; 202. Cloud backup unit; 203. Cold data storage unit;

[0037] 3. Data classification module; 301. Access frequency monitoring unit; 302. Frequency sorting unit; 303. Dynamic standard formulation unit;

[0038] 4. Data migration module; 401. Data classification identification unit; 402. Allocation logic authentication unit; 403. Backup migration unit;

[0039] 5. Traffic-saving analysis module; 501. Migration frequency analysis unit; 502. Fixed update unit. DETAILED DESCRIPTION OF THE INVENTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0041] See also Figures 1-5 , the embodiment of the present invention provides the following technical solutions:

[0042] Embodiment 1: An intelligent data storage management system includes an intelligent data management platform 1 composed of a pooled storage identification module 2, a data classification module 3 and a data migration module 4.

[0043] As a preferred solution, in order to realize classified storage of data and provide security, the pooled storage identification module 2 includes a hot data storage unit 201, a cloud backup unit 202 and a cold data storage unit 203. The hot data storage unit 201 is used to store hot data, that is, data that needs to be accessed quickly. Hot data includes but is not limited to user personal information and transaction record data. The hot data storage unit 201 uses a solid state hard disk as a hot data storage medium, and its read and write speed and response time are better than mechanical hard disks; the cold data storage unit 203 is used to store cold data, that is, data that does not need to be accessed quickly. Cold data includes but is not limited to historical records and archived files. The cold data storage unit 203 uses a mechanical hard disk as a cold data storage medium, and its capacity is better than that of a solid state hard disk, with capacity advantages and price advantages. The cold data storage unit 203 is connected to the hot data storage unit 201 to provide security for the rapid migration of hot data; the hot data storage unit 201 and the cold data storage unit 203 are both connected to the hot data storage unit 201, and the cloud backup unit 202 is used to perform cloud backup of the data stored in the hot data storage unit 201 and the cold data storage unit 203, and to perform regular updates.

[0044] As a preferred solution, the pooled storage identification module 2 is connected to the data classification module 3. The data classification module 3 includes an access frequency monitoring unit 301, a frequency sorting unit 302 and a dynamic standard formulation unit 303. The access frequency monitoring unit 301 is used to calculate the access frequency of different data in a unit time according to the operation log. The unit time is manually formulated. The access frequency is the result of dividing the number of access times of the data in a unit time by the unit time. The access frequency monitoring unit 301 is connected to the frequency sorting unit 302. The frequency sorting unit 302 is used to sort a number of data in order from high to low according to the data access frequency, and obtain the size of the corresponding data volume.

[0045] The frequency sorting unit 302 is connected to the dynamic standard formulation unit 303, which is used to calculate the sum of the corresponding data volumes of the plurality of sequentially connected data that are ranked first, and select a plurality of data combinations that meet the capacity of the hot data storage unit 201 as the hot data combination {a1, a2, ..., a n};

[0046] Among them, the method of screening several data combinations that meet the capacity of the hot data storage unit 201 includes:

[0047] MAX(A1+A2+…+A n )≤T

[0048] In the formula, A1 is the data volume size corresponding to the data a1 with the highest access frequency, and A n is the data volume size corresponding to the data a n with the nth highest access frequency, and T is the capacity of the hot data storage unit 201.

[0049] As a preferred solution, for the intelligent migration management of hot data, the data classification module 3 is docked with the data migration module 4, and the data migration module 4 is docked with the pooling storage identification module 2. The data migration module 4 includes a data classification identification unit 401, an allocation logic authentication unit 402, and a backup migration unit 403. The data classification identification unit 401 is used to perform hot data identification on all data in the hot data combination, locate the current storage location of the data with hot data identification, perform cold migration identification on the data without hot data identification stored in the hot data storage unit 201, and perform hot migration identification on the data with hot data identification stored in the cold data storage unit 203;

[0050] The data classification identification unit 401 is docked with the allocation logic authentication unit 402. The allocation logic authentication unit 402 is used to upload the data with cold migration identification and hot migration identification to the cloud backup unit 202, perform backup update on the corresponding backup data of the data, and delete the data with cold migration identification in the hot data storage unit 201 and the data with hot migration identification in the cold data storage unit 203;

[0051] The allocation logic authentication unit 402 is docked with the backup migration unit 403. The backup migration unit 403 is used to migrate the data with hot migration identification in the cold data storage unit 203 to the hot data storage unit 201 to improve the data migration efficiency, and then download the backup data corresponding to the data with cold migration identification from the cloud backup unit 202 to the cold data storage unit 203.

[0052] In this embodiment, the classified storage and cloud backup of data are realized, and the dynamic update of hot data is realized according to the access frequency of the data, providing a reliable guarantee for improving the data access speed in the way of data migration.

[0053] Embodiment 2, this embodiment is an improvement of the previous embodiment, an intelligent data storage management system, further comprising a flow saving analysis module 5, a pooled storage identification module 2 and a flow saving analysis module 5 are connected, a data migration module 4 and a flow saving analysis module 5 are connected, the flow saving analysis module 5 comprises a migration frequency analysis unit 501 and a fixed update unit 502, the migration frequency analysis unit 501 is used to filter the data with a hot migration identifier that is the same as the data with a cold migration identifier in adjacent unit time, and when the same situation occurs for a continuous set number of times, a resident hot data identifier is added to the data, wherein the continuous set number of times is manually formulated, preferably 3-5 times, and a secondary screening is performed on the method of screening a plurality of data combinations that meet the capacity of the hot data storage unit 201, and the secondary screening method includes:

[0054] MAX(A1+A2+…+A n )≤TA G

[0055] In the formula, A G The data volume corresponding to the data with the resident hot data mark;

[0056] The migration frequency analysis unit 501 is connected to the fixed update unit 502. The fixed update unit 502 is used to update the data with the permanent hot data identifier when the data does not exist in the hot data combination {a1, a2, ..., a n}, cancel the resident hot data mark of the data, wherein the number of consecutive settings is manually formulated, preferably 3-5 times.

[0057] The advantage of the second embodiment over the first embodiment is that it can realize the identification and processing of frequently migrated data and reduce the loss of network traffic.

[0058] When in use, the data is preferably stored in the hot data storage unit 201. After the hot data storage unit 201 is full, the data is stored in the cold data storage unit 203. During the process, the cloud backup unit 202 performs cloud backup of the data stored in the hot data storage unit 201 and the cold data storage unit 203, and performs regular updates;

[0059] After the hot data storage unit 201 is fully stored, the access frequency monitoring unit 301 calculates the access frequency of different data in a unit time according to the operation log, and the frequency sorting unit 302 sorts the data in descending order of the data access frequency, and obtains the size of the corresponding data volume;

[0060] The dynamic standard formulation unit 303 is used to calculate the sum of the corresponding data amounts of the plurality of sequentially connected data that are ranked first, and select a plurality of data combinations that meet the capacity of the hot data storage unit 201, and the specific method includes:

[0061] MAX(A1 + A2 + … + A n ) ≤ T

[0062] wherein, A1 is the data volume corresponding to the data a1 with the highest access frequency, A n is the data volume corresponding to the data a n with the nth highest access frequency, and T is the capacity of the hot data storage unit 201.

[0063] Obtain the hot data combination {a1, a2, …, a n}, which is the data to be stored in the hot data storage unit 201. Conversely, the remaining data is stored in the cold data storage unit 203.

[0064] The specific data migration method includes:

[0065] The data classification and identification unit 401 performs hot data identification on all data in the hot data combination, locates the current storage location of the data with hot data identification. For the data without hot data identification stored in the hot data storage unit 201, cold migration identification is performed. For the data with hot data identification stored in the cold data storage unit 203, hot migration identification is performed. The allocation and logical authentication unit 402 uploads the data with cold migration identification and hot migration identification to the cloud backup unit 202, updates the backup of the corresponding backup data of the data, and deletes the data with cold migration identification in the hot data storage unit 201 and the data with hot migration identification in the cold data storage unit 203. The backup migration unit 403 migrates the data with hot migration identification in the cold data storage unit 203 to the hot data storage unit 201, and then downloads the backup data corresponding to the data with cold migration identification from the cloud backup unit 202 to the cold data storage unit 203 to complete the data migration.

[0066] Further explanation: The migration frequency analysis unit 501 is used to screen out the data with hot migration identification that is the same as the data with cold migration identification in adjacent unit time. When the same situation occurs continuously for a set number of times, a resident hot data identification is added to the data, and a secondary screening is performed on the method of screening several data combinations that meet the capacity of the hot data storage unit 201. The secondary screening method includes:

[0067] MAX(A1 + A2 + … + A n ) ≤ T - A G

[0068] wherein, A G is the data volume corresponding to the data with resident hot data identification;

[0069] At this time, the data with the resident hot data flag resides in the hot data storage unit 201. To prevent this data from occupying the hot data storage unit 201 invalidly for a long time, when the data with the resident hot data flag does not exist in the hot data combination {a1, a2, …, a n} for a continuous set number of times, the fixed update unit 502 cancels the resident hot data flag of this data.

[0070] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent data storage management system, comprising an intelligent data management platform (1), characterized in that: The intelligent data management platform (1) comprises a pooled storage identification module (2), a data classification module (3), a data migration module (4) and a flow saving analysis module (5); the pooled storage identification module (2) is connected to the data classification module (3) and the flow saving analysis module (5) respectively; the data classification module (3) is connected to the data migration module (4); the data migration module (4) is connected to the pooled storage identification module (2); and the data migration module (4) is connected to the flow saving analysis module (5).

2. An intelligent data storage management system according to claim 1, characterized in that: The pooled storage identification module (2) comprises a hot data storage unit (201), a cloud backup unit (202) and a cold data storage unit (203); the hot data storage unit (201) is connected to the cloud backup unit (202), the cloud backup unit (202) is connected to the cold data storage unit (203), and the cold data storage unit (203) is connected to the hot data storage unit (201).

3. An intelligent data storage management system according to claim 2, characterized in that: The thermal data storage unit (201) is used to store thermal data; The cold data storage unit (203) is used to store cold data; The cloud backup unit (202) is used to perform cloud backup of the data stored in the hot data storage unit (201) and the cold data storage unit (203), and to perform regular updates.

4. The intelligent data storage management system according to claim 3, characterized in that: The data classification module (3) comprises an access frequency monitoring unit (301), a frequency sorting unit (302) and a dynamic standard formulation unit (303), wherein the access frequency monitoring unit (301) is connected to the frequency sorting unit (302), and the frequency sorting unit (302) is connected to the dynamic standard formulation unit (303).

5. The intelligent data storage management system according to claim 4, characterized in that: The access frequency monitoring unit (301) is used to calculate the access frequency of different data in a unit time according to the operation log; The frequency sorting unit (302) is used to sort a number of data in order of data access frequency from high to low, and obtain the size of the corresponding data volume; The dynamic standard formulation unit (303) is used to calculate the sum of the corresponding data sizes of a plurality of sequentially connected data that are ranked first, and select a plurality of data combinations that meet the capacity of the hot data storage unit (201) as the hot data combination {a1, a2, ..., a n }.

6. An intelligent data storage management system according to claim 5, characterized in that: The method of screening a plurality of data combinations that meet the capacity of the hot data storage unit (201) includes: <h2 style=";text-align:left;direction:ltr">MAX(A1+A2+…+A<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> )≤T In the formula, A1 is the data size corresponding to the most frequently accessed data a1, A n The data a of the nth access frequency n The corresponding data volume size, T is the capacity of the hot data storage unit (201).

7. An intelligent data storage management system according to claim 6, characterized in that: The data migration module (4) comprises a data classification identification unit (401), an allocation logic authentication unit (402) and a backup migration unit (403); the data classification identification unit (401) is connected to the allocation logic authentication unit (402), and the allocation logic authentication unit (402) is connected to the backup migration unit (403).

8. An intelligent data storage management system according to claim 7, characterized in that: The data classification identification unit (401) is used to perform hot data identification on all data in the hot data combination, locate the current storage position of the data with the hot data identification, perform cold migration identification on the data without the hot data identification stored in the hot data storage unit (201), and perform hot migration identification on the data with the hot data identification stored in the cold data storage unit (203); The allocation logic authentication unit (402) is used to upload the data with the cold migration mark and the hot migration mark to the cloud backup unit (202), perform backup update on the backup data corresponding to the data, and delete the data with the cold migration mark in the hot data storage unit (201) and the data with the hot migration mark in the cold data storage unit (203); The backup migration unit (403) is used to migrate the data with a hot migration identifier in the cold data storage unit (203) to the hot data storage unit (201), and then download the backup data corresponding to the data with the cold migration identifier from the cloud backup unit (202) to the cold data storage unit (203).

9. The intelligent data storage management system according to claim 8, characterized in that: The flow saving analysis module (5) comprises a migration frequency analysis unit (501) and a fixed update unit (502), and the migration frequency analysis unit (501) is connected to the fixed update unit (502).

10. The intelligent data storage management system according to claim 9, characterized in that: The migration frequency analysis unit (501) is used to screen data with hot migration marks that are the same as data with cold migration marks in adjacent unit time, and when the same situation occurs for a set number of consecutive times, a permanent hot data mark is added to the data, and a secondary screening is performed on a plurality of data combinations that meet the capacity of the hot data storage unit (201), and the secondary screening method includes: MAX(A1+A2+…+A n )≤T-A G In the formula, A G The data volume corresponding to the data with the resident hot data mark; The fixed updating unit (502) is used to update the data with the permanent hot data identifier after it does not exist in the hot data combination {a1, a2, ..., a n }, cancel the resident hot data flag of the data.

Images