An intelligent identification and optimization system for data life cycle

Abstract

The present invention discloses an intelligent recognition and optimization system for data life cycle, comprising a storage management module and a policy management module; the storage management comprises an analysis module and a governance module, the analysis module evaluates the storage health score of the system by analyzing the number of small files and cold data capacity of the file system, as well as the health of the storage nodes; the governance module specifies corresponding storage policies according to the health score, optimizes storage through a migration tool, and then comprehensively grasps the storage and governance status through statistical charts; the policy management module supports the management of hierarchical storage policies, analysis policies and compression policies, and users set hierarchical storage policies and compression policies for directories to optimize file storage; analysis policies are set for small files and cold data to assist in data analysis; the present invention provides an intelligent recognition and optimization system for data life cycle that can understand the health status of each directory and even file and optimize storage.

Description

Technical Field

[0001] The present invention relates to the field of computer storage, and more particularly to an intelligent identification and optimization system for data life cycle. Background Art

[0002] In the process of enterprise big data application, the storage space occupied by data such as HDFS is getting larger and larger, resulting in lower operating efficiency. At the same time, enterprises cannot fully control the usage of all data, and they also encounter many difficulties in data optimization and have no idea where to start.

[0003] As enterprise big data clusters grow in use over time and accumulate more data, not only does this increase memory usage and read and write times, it also hinders cluster expansion and reduces operational efficiency. Therefore, understanding the overall status of data files, accurately identifying directories and data files that require optimization, and managing this data become crucial. Summary of the Invention

[0004] The present invention overcomes the deficiencies of the prior art and provides an intelligent recognition and optimization system for data lifecycle that can understand the health status of each directory and even files and optimize storage.

[0005] The technical solutions of the present invention are as follows:

[0006] An intelligent identification and optimization system for data life cycle, including a storage management module and a policy management module;

[0007] Storage management includes an analysis module and a governance module. The analysis module evaluates the system's storage health score by analyzing the number of small files and cold data capacity in the file system, as well as the health of storage nodes. The governance module specifies storage policies based on the health score, optimizes storage through migration tools, and provides a comprehensive overview of storage and governance status through statistical charts.

[0008] The policy management module supports the management of tiered storage policies, analysis policies, and compression policies. Users can set tiered storage policies and compression policies for directories to optimize file storage; and set analysis policies for small files and cold data to facilitate data analysis.

[0009] The underlying technology framework includes MySQL, Hive, and HDFS. Hive Client is used to connect to Hive, WebHDFS, and dfsadmin are used to access HDFS to obtain Hive and HDFS data. MyBatis is used to interact with MySQL for data storage.

[0010] The specific steps are as follows:

[0011] 101) Metadata acquisition step: Obtain HDFS metadata by analyzing fsimage;

[0012] 102) Metadata indexing step: parsing the metadata file obtained in step 101) and constructing it into a multi-tree structure;

[0013] 103) Data analysis step: Count the number and size of all files in the directory and the number and size of different data types, perform total statistics, ranking analysis, and proportion analysis to obtain a storage health score;

[0014] 104) Data strategy configuration steps: including tiered storage strategy, analysis strategy and compression strategy; tiered storage strategy is a heterogeneous storage strategy that stores data on different storage media according to the popularity of data access, so that HDFS storage can flexibly and efficiently cope with various application scenarios; the analysis strategy sets the user's definition of small files, the setting of the threshold for the number of small files, the definition of cold data and the threshold for the total amount of cold data, the setting of the disk capacity threshold, and the setting of the scheduling time threshold for the system to perform analysis; the compression strategy sets the erasure code to view all currently available erasure codes to ensure data migration, and view and record the migration log.

[0015] Furthermore, the middle layer of the overall technical framework uses Schedule to implement periodic scheduling and constructs a multi-branch tree to facilitate data analysis; the upper layer of the overall technical framework provides an external API call interface and a visual UI operation interface.

[0016] Furthermore, metadata includes: Path-directory path, Replication-number of backups, ModificationTime-last modification time, AccessTime-last access time, PreferredBlockSize-preferred block size, BlocksCount-number of blocks, FileSize-file size, NSQUOTA-name quota, DSQUOTA-space quota, Permission-permission, UserName-user, and GroupName-user group;

[0017] Specifically, it obtains fsimage, parses fsimage into metadata in a specified format, and finally outputs the oiv file.

[0018] Furthermore, data analysis includes small file analysis, cold data analysis, hot data analysis, table analysis, damaged block analysis, and disk memory analysis:

[0019] Small file analysis is used to count the number and size of small files based on policy settings.

[0020] Cold data analysis is used to count the number and size of cold data based on policy settings.

[0021] Hot data analysis is used to count the number and size of hot data based on policy settings.

[0022] Table analysis is used to count the number and size of small files in all tables in the database based on policy settings.

[0023] Corrupted block analysis is used to count the number of corrupted file blocks.

[0024] Disk memory analysis is used to count the total amount and usage of disks.

[0025] Furthermore, the scoring rules for storage health score include disk score, small file score, cold data score and file block score;

[0026] The total disk score in the disk score is 30. Assuming the number of nodes is n, the score of each node is When the disk usage of w1 nodes exceeds the threshold, the points; assuming the node has m disks, the score of each disk is When the total disk storage does not exceed the threshold, but a single disk exceeds the threshold, the points will be deducted. If w2 disks exceed the threshold, the points will be deducted. point;

[0027] The total score of small files in the small file score is 30, and the threshold number of small files is set to t. When the number of small files x exceeds the threshold by 1-10%, 1 point is deducted. When it exceeds by 11-20%, another 1 point is deducted, and so on;

[0028] The total score of cold data is 30. If there are 100GB of unprocessed cold data, that is, data without tiered storage strategy or erasure coding strategy, 1 point will be deducted for every 100GB, until the total score is deducted.

[0029] The total score of the file block is 10. If z file blocks are damaged, one point will be deducted for every 10 damaged blocks (i.e., 1-10 damaged blocks). One point will be deducted for every 11-20 damaged blocks, and this will continue until the total score is deducted.

[0030] Therefore, the calculation formula of the storage health score S obtained above is as follows:

[0031]

[0032] The points deducted for each item cannot exceed the total score of all items.

[0033] Furthermore, HDFS supports a variety of common storage types, including:

[0034] ARCHIVE: A storage medium with high storage density but low power consumption, used to store cold data;

[0035] DISK: disk medium, which is the default storage medium of HDFS;

[0036] SSD: solid-state drive storage media;

[0037] RAM_DISK: The data is written to the memory and a copy is asynchronously written to the storage medium.

[0038] Furthermore, tiering policies include PROVIDED, COLD, WARM, HOT, ONE_SSD, ALL_SSD, and LAZY_PERSIST;

[0039] PROVIDED is external HDFS storage, and the storage medium is DISK;

[0040] COLD means all copies are stored on archive storage, and the storage medium is ARCHIVE;

[0041] WARM uses one copy to be saved on disk, and the remaining copies are saved on archive storage, with the storage media being DISK and ARCHIVE;

[0042] HOT uses the default storage strategy of storing all copies on disk, and the storage medium is DISK.

[0043] ONE_SSD uses one copy stored in the SSD and the remaining copies stored in disks. The storage media is SSD and disk.

[0044] ALL_SSD: All copies are stored in SSD, and the storage medium is SSD;

[0045] LAZY_PERSIST uses one copy to be stored in the memory RAM_DISK, and the remaining copies are stored in the disk RAM_DISK. The storage medium is DISK.

[0046] The advantages of the present invention are:

[0047] This invention not only captures the overall health of the system or the status of a specific directory, but also understands the health of individual directories and even files. It accurately determines the distribution of specific small files and counts the directories with the most small files. It also manages data and optimizes storage through migration tools. Based on configured data analysis strategies, it intelligently analyzes hot and cold data and displays statistics. It supports statistical analysis and management of Hive tables. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a product architecture diagram of the present invention;

[0049] Figure 2 is an operational flow chart of the present invention;

[0050] Figure 3 It is a technical framework diagram of the present invention;

[0051] Figure 4 It is a technical flow chart of the present invention;

[0052] Figure 5 A flowchart for obtaining metadata of the present invention;

[0053] Figure 6 A design diagram for the metadata index of the present invention;

[0054] Figure 7 This is a diagram of the storage health score structure of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present invention and are not intended to limit the present invention. Parts not specifically described in this solution may be implemented using conventional technical means.

[0056] The following are explanations of the nouns that may be involved:

[0057] HDFS is the Hadoop distributed file system. Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. Small files refer to files whose size is significantly smaller than the block size on HDFS (64MB by default, 128MB by default in Hadoop 2.x). Cold data refers to data that is infrequently accessed, not frequently accessed, or never accessed, but still needs to be retained for a long time. Hot data refers to very popular and frequently accessed data. Fsimage is a file in HDFS that stores information about all directories and files in the entire HDFS file system and is loaded when HDFS starts. Oiv is a file format, and its full name is offline image viewer. Erasure coding, also known as EC, is a data protection technology. It can replace multiple copies and use less storage to ensure the same level of fault tolerance.

[0058] like Figures 1 to 7 The intelligent identification and optimization system for data lifecycle includes a storage management module and a policy management module, and mainly supports storage, management, analysis and optimization of files (HDFS) and tables (HIVE).

[0059] Storage management includes an analysis module and a governance module. The analysis module assesses the system's storage health score by analyzing the number of small files and cold data capacity in the file system, as well as the health of storage nodes. The governance module then assigns storage policies based on the health score, optimizes storage through migration tools, and provides a comprehensive overview of storage and governance status through statistical charts.

[0060] The policy management module supports the management of tiered storage policies, analysis policies, and compression policies. Users can set tiered storage policies and compression policies for directories to optimize file storage. Analysis policies can also be set for small files and cold data to facilitate data analysis.

[0061] That is, the local cache is built by obtaining fsimage. The client sends an analysis request, and the policy configuration is issued based on the analysis results. Finally, the issued policy is made effective through data migration.

[0062] The underlying technical framework includes MySQL, Hive, and HDFS. Hive Client is used to connect to Hive, WebHDFS, and dfsadmin are used to access HDFS to retrieve data from Hive and HDFS. MyBatis is used to interact with MySQL for data storage. The middle layer of the overall technical framework uses Schedule to implement periodic scheduling and construct a multi-tree tree to facilitate data analysis. The upper layer of the overall technical framework provides an external API call interface and a visual user interface.

[0063] The specific steps are as follows:

[0064] 101) Metadata Acquisition Step: HDFS metadata is obtained by analyzing fsimage. The metadata includes: Path (directory path), Replication (number of replicas), ModificationTime (last modification time), AccessTime (last access time), PreferredBlockSize (preferred block size in bytes), BlocksCount (number of blocks), FileSize (file size in bytes), NSQUOTA (name quota (limits the number of files and directories allowed in a specified directory), DSQUOTA (space quota (limits the number of bytes allowed in a directory), Permission (permissions), UserName (user), and GroupName (user group).

[0065] Specifically, it obtains fsimage, parses fsimage into metadata in a specified format, and finally outputs the oiv file.

[0066] 102) Metadata indexing step: Parse the metadata file obtained in step 101) and construct it into a multi-branch tree structure.

[0067] 103) Data Analysis Step: Count the number and size of all files in the directory and the number and size of different data types, perform total statistics, ranking analysis, and proportion analysis to obtain a storage health score. Specific data analysis includes small file analysis, cold data analysis, hot data analysis, table analysis, damaged block analysis, and disk memory analysis:

[0068] Small file analysis is used to count the number and size of small files based on policy settings.

[0069] Cold data analysis is used to count the number and size of cold data based on policy settings.

[0070] Hot data analysis is used to count the number and size of hot data based on policy settings.

[0071] Table analysis is used to count the number and size of small files in all tables in the database based on policy settings.

[0072] Corrupted block analysis is used to count the number of corrupted file blocks.

[0073] Disk memory analysis is used to count the total amount and usage of disks.

[0074] The scoring rules for storage health score include disk score, small file score, cold data score, and file block score.

[0075] The total disk score in the disk score is 30. Assuming the number of nodes is n, the score of each node is When the disk usage of w1 nodes exceeds the threshold, the Suppose the node has m disks, then the score of each disk is When the total disk storage does not exceed the threshold, but a single disk exceeds the threshold, the points will be deducted. If w2 disks exceed the threshold, the points will be deducted. point.

[0076] The total score of small files in the small file score is 30, and the threshold number of small files is set to t. When the number of small files x exceeds the threshold by 1-10%, 1 point is deducted. When it exceeds by 11-20%, another 1 point is deducted, and so on.

[0077] The total score of cold data in the cold data score is 30. If there are yG of unprocessed cold data, that is, data without a tiered storage strategy or erasure code strategy, 1 point will be deducted for every 100G, until the total score is deducted.

[0078] The total score of the file blocks in the file block score is 10. It is assumed that if z file blocks are damaged, one point will be deducted for every 10 damaged blocks, that is, if blocks 1-10 are damaged, and another point will be deducted for blocks 11-20, until the total score is deducted.

[0079] Therefore, the calculation formula of the storage health score S obtained above is as follows:

[0080]

[0081] The points deducted for each item cannot exceed the total score of all items.

[0082] 104) Data strategy configuration steps: including tiered storage strategy, analysis strategy and compression strategy.

[0083] A tiered storage strategy, also known as a heterogeneous storage strategy, stores data on different storage media based on data access popularity, enabling HDFS storage to flexibly and efficiently address various application scenarios. Data tiering is based on HDFS's support for heterogeneous storage and the ability to configure heterogeneous storage strategies.

[0084] HDFS supports a variety of common storage types, including:

[0085] ARCHIVE: A storage medium with high storage density but low power consumption, used to store cold data.

[0086] DISK: Disk media, which is the default storage medium of HDFS.

[0087] SSD: Solid-state drive storage media.

[0088] RAM_DISK: The data is written to the memory and a copy is asynchronously written to the storage medium.

[0089] Furthermore, tiering policies include PROVIDED, COLD, WARM, HOT, ONE_SSD, ALL_SSD, and LAZY_PERSIST.

[0090] PROVIDED is external HDFS storage, and the storage medium is DISK.

[0091] COLD means all copies are stored on archive storage, and the storage medium is ARCHIVE.

[0092] WARM uses one copy to be saved on disk, and the remaining copies are saved on archive storage. The storage media are DISK and ARCHIVE.

[0093] HOT uses the default storage strategy of storing all copies on disk, and the storage medium is DISK.

[0094] ONE_SSD uses SSD and disks to store one copy and the remaining copies.

[0095] ALL_SSD: All replicas are stored in SSD, and the storage medium is SSD.

[0096] LAZY_PERSIST uses one copy to be stored in the memory RAM_DISK, and the remaining copies are stored in the disk RAM_DISK. The storage medium is DISK.

[0097] The analysis strategy sets the user's definition of small files, the threshold for the number of small files, the definition of cold data and the threshold for the total amount of cold data, the threshold for disk capacity, and the scheduling time threshold for system analysis execution.

[0098] Compression policy settings for erasure codes allow you to view all currently available erasure codes to ensure data migration, and view and record migration logs. Specific erasure codes include: RS-10-4-1024k, RS-3-2-1024k, RS-6-3-1024k, RS-LEGACY-6-3-1024k, and XOR-2-1-1024k.

[0099] In summary, the present invention intelligently identifies the entire data lifecycle, including small files and cold data. Based on statistical and analytical data, it assesses storage health scores and intelligently manages, compresses, and migrates data. A visual user interface helps users manage and govern data clearly and intuitively, providing APIs for cluster statistics, data analysis, data governance, data migration, and file management.

[0100] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the concept of the present invention. These improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An intelligent identification and optimization system for data lifecycle, characterized by: Includes storage management module and policy management module; Storage management includes analysis modules and governance modules. The analysis module evaluates the system's storage health score by analyzing the number of small files and cold data capacity in the file system, as well as the health of storage nodes. The governance module specifies storage policies based on health scores, optimizes storage through migration tools, and provides a comprehensive overview of storage and governance status through statistical charts. The policy management module supports the management of tiered storage policies, analysis policies, and compression policies. Users can set tiered storage policies and compression policies for directories to optimize file storage; and set analysis policies for small files and cold data to facilitate data analysis. The underlying technology framework includes MySQL, Hive, and HDFS. Hive Client is used to connect to Hive, WebHDFS, and dfsadmin are used to access HDFS to obtain Hive and HDFS data. MyBatis is used to interact with MySQL for data storage. The specific steps are as follows: 101) Metadata acquisition step: Obtain HDFS metadata by analyzing fsimage; 102) Metadata indexing step: parsing the metadata file obtained in step 101) and constructing it into a multi-tree structure; 103) Data analysis step: Count the number and size of all files in the directory and the number and size of different data types, perform total statistics, ranking analysis, and proportion analysis to obtain a storage health score; Data analysis includes small file analysis, cold data analysis, hot data analysis, table analysis, damaged block analysis, and disk memory analysis. Cold data refers to data for which no tiered storage strategy or erasure coding strategy is set. The calculation formula for storage health score S is as follows: Among them, the deduction score for each item cannot exceed the total score of each item; n is the number of nodes, is the score for each node, w1 is the number of disk nodes whose usage exceeds the threshold, m is the number of disks, is the score of each disk, w2 is the number of disks exceeding the threshold; y is the size of unprocessed cold data, 1 point is deducted for every 100G; z is the number of damaged file blocks, 1 point is deducted for every 10 damaged blocks; t is the number of small files threshold; 104) Data strategy configuration steps: including tiered storage strategy, analysis strategy and compression strategy; tiered storage strategy is a heterogeneous storage strategy that stores data on different storage media according to the popularity of data access, so that HDFS storage can flexibly and efficiently cope with various application scenarios; the analysis strategy sets the user's definition of small files, the setting of the threshold for the number of small files, the definition of cold data and the threshold for the total amount of cold data, the setting of the disk capacity threshold, and the setting of the scheduling time threshold for the system to perform analysis; the compression strategy sets the erasure code to view all currently available erasure codes to ensure data migration, and view and record the migration log.

2. The intelligent identification and optimization system for data lifecycle according to claim 1, characterized in that: The middle layer of the overall technical framework uses Schedule to implement periodic scheduling and constructs a multi-branch tree to facilitate data analysis; the upper layer of the overall technical framework provides an external API call interface and a visual UI operation interface.

3. The intelligent identification and optimization system for data lifecycle according to claim 1, characterized in that: Metadata includes: Path-directory path, Replication-number of backups, ModificationTime-last modification time, AccessTime-last access time, PreferredBlockSize-preferred block size, BlocksCount-number of blocks, FileSize-file size, NSQUOTA-name quota, DSQUOTA-space quota, Permission-permission, UserName-user and GroupName-user group; Specifically, it obtains fsimage, parses fsimage into metadata in a specified format, and finally outputs the oiv file.

4. The intelligent identification and optimization system for data lifecycle according to claim 1, characterized in that: Small file analysis is used to count the number and size of small files according to policy settings; Cold data analysis is used to count the number and size of cold data based on policy settings; Hot data analysis is used to count the number and size of hot data according to policy settings; Table analysis is used to count the number and size of small files in all tables in the database according to policy settings; Damaged block analysis is used to count the number of damaged file blocks; Disk memory analysis is used to count the total amount and usage of disks.

5. The intelligent identification and optimization system for data lifecycle according to claim 1, characterized in that: HDFS supports a variety of common storage types, including: ARCHIVE: A storage medium with high storage density but low power consumption, used to store cold data; DISK: disk medium, which is the default storage medium of HDFS; SSD: solid-state drive storage media; RAM_DISK: The data is written to the memory and a copy is asynchronously written to the storage medium.

6. The intelligent identification and optimization system for data lifecycle according to claim 5, characterized in that: The tiering policies include PROVIDED, COLD, WARM, HOT, ONE_SSD, ALL_SSD, and LAZY_PERSIST; PROVIDED is external HDFS storage, and the storage medium is DISK; COLD means all copies are stored on archive storage, and the storage medium is ARCHIVE; WARM uses one copy to be saved on disk, and the remaining copies are saved on archive storage, with the storage media being DISK and ARCHIVE; HOT uses the default storage strategy of storing all copies on disk, and the storage medium is DISK. ONE_SSD uses one copy stored in the SSD and the remaining copies stored in disks. The storage media is SSD and disk. ALL_SSD: All copies are stored in SSD, and the storage medium is SSD; LAZY_PERSIST uses one copy to be stored in the memory RAM_DISK, and the remaining copies are stored in the disk RAM_DISK. The storage medium is DISK.

Images