Data management method and device based on cold data migration, equipment and storage medium

Abstract

The invention relates to the technical field of data management and the field of financial science and technology, and discloses a data management method based on cold data migration, which comprises the following steps: extracting metadata in a distributed file system, and importing the metadata into a metadata storage library of a distributed data processing platform. And analyzing file information in the metadata storage library through an offline analysis task, identifying cold data, and storing file path information of the cold data in a database. And according to the identified cold data, generating a data migration plan comprising a migration target position, migration priority ranking and migration time arrangement. And migrating the cold data from the original storage position to the target far-end storage, and updating a table partition path in the metadata storage library to point to a new storage position. And after migration is completed, deleting the cold data in the original storage position. According to the method, the cold data and the computing resources are effectively separated, the utilization efficiency of the storage resources is optimized, the overall resource consumption of the system is reduced, and meanwhile the expansibility and the performance of the system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data management method, device, equipment and storage medium based on cold data migration. Background Art

[0002] In the financial sector, with the rapid development of business and the explosive growth of data volume, financial institutions are faced with the challenge of how to efficiently store and process massive amounts of data. To address this problem, many financial institutions have adopted the distributed data processing framework Hadoop. As an extensible software framework that can perform distributed processing of large amounts of data, Hadoop is widely used in data analysis, risk management, customer behavior prediction and other application scenarios in the financial sector.

[0003] The Hadoop framework contains several key components, among which HDFS (Hadoop Distributed File System) is used to provide efficient distributed storage for massive data, and YARN (Yet Another Resource Negotiator) is responsible for providing resource management and scheduling services for various computing frameworks. In traditional Hadoop cluster deployment, in order to maximize the locality of computing resources, computing and storage nodes are usually deployed on the same machine. In this way, the overhead of data transmission can be reduced and the efficiency of data processing can be improved.

[0004] However, this deployment method faces some practical problems in the financial industry. As business needs increase, financial institutions need to frequently expand the size of Hadoop clusters to meet growing computing and storage needs. Generally, it is relatively easy to improve system performance by adding machines to horizontally expand the Hadoop cluster. However, this expansion method requires that the host's computing and storage resources achieve a relatively balanced configuration, such as the configuration of CPU, memory, network card, and storage resources need to match each other. Since these resources must be expanded at the same time, the overall cost of a single machine is high, and in some cases, resources are wasted.

[0005] Specifically, the data stored in HDFS follows the 80 / 20 rule, that is, 80% of the data is cold data with extremely low access frequency. Although these cold data occupy a large amount of storage space, due to the low access frequency, the machines storing these data have low demand for computing resources such as CPU and memory in actual operation, resulting in a waste of computing resources. In the financial industry, this problem of resource waste is particularly prominent, because cold data occupies a large amount of expensive computing resources, but these resources are not fully utilized, affecting the overall resource utilization efficiency.

[0006] Therefore, how to optimize the utilization of computing and storage resources and reduce resource waste while ensuring efficient data processing has become an urgent problem that the financial industry needs to solve when using the Hadoop framework for big data processing. Summary of the invention

[0007] The main purpose of the present invention is to provide a data management method, device, equipment and storage medium based on cold data migration, aiming to solve the technical problem that the existing technology cannot effectively identify and manage cold data in a distributed file system, resulting in cold data occupying computing resources, causing system resource waste and poor performance.

[0008] To achieve the above object, the present invention provides a data management method based on cold data migration, comprising:

[0009] Extracting metadata from a distributed file system, and importing the metadata into a metadata repository of a distributed data processing platform;

[0010] By means of an offline analysis task of a distributed data processing platform, file information of metadata in the metadata repository is analyzed, cold data is identified, and file path information of the cold data is stored in a database;

[0011] Based on the identified cold data, generate a data migration plan, the data migration plan including a migration target location, a migration priority ranking, and a migration schedule;

[0012] According to the data migration plan, migrate the cold data from the original storage location to the target remote storage, and update the table partition path in the metadata repository to point to the new storage location after the migration;

[0013] After completing the data migration, delete the migrated cold data in the original storage location.

[0014] In one embodiment, analyzing file information of metadata in the metadata repository and identifying cold data includes:

[0015] Analyzing the metadata in the metadata repository to extract file information related to the file, the file information including file path, last access time, last modification time, file size and creation time;

[0016] Set access frequency thresholds and time point thresholds for identifying cold data;

[0017] Based on the file information, count the file access frequency, compare the file access frequency with the access frequency threshold, and if the file access frequency is lower than the access frequency threshold, mark the file as cold data;

[0018] Compare the last access time of the file with a time point threshold, and if the last access time of the file is earlier than the time point threshold, mark the file as cold data;

[0019] Files are graded according to their size and comprehensively evaluated based on the file access frequency and last access time. If the comprehensive evaluation criteria are met, the file is marked as cold data.

[0020] In one embodiment, extracting metadata from a distributed file system and importing the metadata into a metadata repository of a distributed data processing platform includes:

[0021] Export the current FSImage file from the distributed file system, wherein the FSImage file contains metadata of all files and directories in the file system;

[0022] Convert the exported FSImage file into a file format that complies with the table structure specification of the distributed data processing platform, wherein the conversion includes converting the metadata into a structured table format;

[0023] Selecting a target table in a distributed data processing platform, and importing the converted metadata file into a partition of the target table according to a specified partitioning rule, wherein the partitioning rule is based on a date or a file type;

[0024] The imported metadata files are stored in the metadata repository of the distributed data processing platform for use in data analysis tasks.

[0025] In one embodiment, based on the identified cold data, a data migration plan is generated, including:

[0026] Determine a migration target location for cold data, wherein the migration target location includes remote storage, cloud storage, or an archive storage device;

[0027] Calculate the migration priority of cold data based on file size, last access time and access frequency;

[0028] Divide cold data into multiple migration batches according to migration priority, and set a migration schedule for each migration batch;

[0029] Generate a data migration plan based on the migration target location, migration priority, and migration schedule of the cold data.

[0030] In one embodiment, the migration priority of cold data is calculated based on file size, last access time, and access frequency, including:

[0031] The weights of file size, last access time and access frequency are set according to the amount of storage resources occupied by file size, the reflection of last access time on data usage frequency and the data usage reflected by access frequency;

[0032] Standardize the data of file size, last access time and access frequency, and convert data of different scales into a unified numerical range;

[0033] Based on the standardized file size, last access time, and access frequency data, the migration priority score of each cold data file is calculated according to the set weight. The priority score is obtained by weighting the file size, last access time, and access frequency according to their respective weights.

[0034] Sort the cold data files according to their priority scores and assign corresponding migration orders to the cold data files.

[0035] In one embodiment, after deleting the migrated cold data in the original storage location, the method further includes:

[0036] Monitor the access to migrated cold data and record the access time and frequency of cold data files;

[0037] Determine whether cold data becomes active again based on the preset activity threshold;

[0038] If the access frequency of cold data reaches the set active threshold, a data migration plan is generated;

[0039] Execute the data migration plan to migrate the reactivated cold data to the original storage location or high-frequency storage area;

[0040] Update the file path information in the metadata repository and update the new storage location of the migrated file to the corresponding table partition record;

[0041] After the data is migrated back, the cache content is updated and the specific information of the migration operation is recorded.

[0042] In one embodiment, according to the data migration plan, migrating the cold data from the original storage location to the target remote storage includes:

[0043] Read the data migration plan to obtain a cold data file list, a migration target location, a migration priority, and a migration schedule;

[0044] Check the availability of the migration target location, verify whether the storage space meets the migration requirements, and verify the stability of the network connection;

[0045] Start the migration tasks of cold data files in sequence according to the migration priority and migration schedule in the data migration plan;

[0046] During the migration process, cold data files that exceed the preset size threshold are split and transferred part by part;

[0047] Monitor the transmission progress, transmission rate and stability of network connection in real time, and record the transmission data and status;

[0048] After completing the data migration, perform data verification to check whether the data is intact during the transmission process, and record the log information of the migration process in detail.

[0049] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a data management device based on cold data migration, wherein the data management device based on cold data migration includes a memory, a processor, and a data management program based on cold data migration stored in the memory and executable on the processor, wherein the data management program based on cold data migration, when executed by the processor, implements the steps of the data management method based on cold data migration as described above.

[0050] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a data management program based on cold data migration is stored, and when the data management program based on cold data migration is executed by a processor, the steps of the data management method based on cold data migration as described above are implemented.

[0051] Beneficial effects: The present invention relates to a data management method based on cold data migration, which extracts metadata from a distributed file system and imports it into a metadata repository of a distributed data processing platform. Through offline analysis tasks, the file information in the metadata repository is analyzed, cold data is identified, and its file path information is stored in a database. Based on the identified cold data, a data migration plan including a migration target location, migration priority sorting, and migration schedule is generated. According to the plan, the cold data is migrated from the original storage location to the target remote storage, and the table partition path in the metadata repository is updated to point to the new storage location. After the migration is completed, the cold data in the original storage location is deleted. The present invention realizes the effective separation of cold data and computing resources, optimizes the utilization efficiency of storage resources, reduces the overall resource consumption of the system, and improves the scalability and performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0053] Figure 1 It is a flow chart of an embodiment of a data management method based on cold data migration of the present invention;

[0054] Figure 2It is a functional module diagram of a preferred embodiment of a data management device based on cold data migration of the present invention;

[0055] Figure 3 It is a structural schematic diagram of the device hardware operating environment involved in the data management device embodiment scheme based on cold data migration of the present invention. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0057] In the financial sector, with the rapid development of business and the explosive growth of data volume, financial institutions are faced with the challenge of how to efficiently store and process massive amounts of data. To address this problem, many financial institutions have adopted the distributed data processing framework Hadoop. As an extensible software framework that can perform distributed processing of large amounts of data, Hadoop is widely used in data analysis, risk management, customer behavior prediction and other application scenarios in the financial sector.

[0058] The Hadoop framework contains several key components, among which HDFS (Hadoop Distributed File System) is used to provide efficient distributed storage for massive data, and YARN (Yet Another Resource Negotiator) is responsible for providing resource management and scheduling services for various computing frameworks. In traditional Hadoop cluster deployment, in order to maximize the locality of computing resources, computing and storage nodes are usually deployed on the same machine. In this way, the overhead of data transmission can be reduced and the efficiency of data processing can be improved.

[0059] However, this deployment method faces some practical problems in the financial industry. As business needs increase, financial institutions need to frequently expand the size of Hadoop clusters to meet growing computing and storage needs. Generally, it is relatively easy to improve system performance by adding machines to horizontally expand the Hadoop cluster. However, this expansion method requires that the host's computing and storage resources achieve a relatively balanced configuration, such as the configuration of CPU, memory, network card, and storage resources need to match each other. Since these resources must be expanded at the same time, the overall cost of a single machine is high, and in some cases, resources are wasted.

[0060] Specifically, the data stored in HDFS follows the 80 / 20 rule, that is, 80% of the data is cold data with extremely low access frequency. Although these cold data occupy a large amount of storage space, due to the low access frequency, the machines storing these data have low demand for computing resources such as CPU and memory in actual operation, resulting in a waste of computing resources. In the financial industry, this problem of resource waste is particularly prominent, because cold data occupies a large amount of expensive computing resources, but these resources are not fully utilized, affecting the overall resource utilization efficiency.

[0061] Therefore, how to optimize the utilization of computing and storage resources and reduce resource waste while ensuring efficient data processing has become an urgent problem that the financial industry needs to solve when using the Hadoop framework for big data processing.

[0062] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a data management method based on cold data migration provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0063] like Figure 1 As shown, the data management method based on cold data migration proposed by the present invention includes the following steps:

[0064] S10, extracting metadata from the distributed file system, and importing the metadata into a metadata repository of the distributed data processing platform;

[0065] In this embodiment, the FSImage file of the current system is obtained from a distributed file system (such as HDFS). The FSImage file is a metadata snapshot of the file system, which contains metadata such as the structure, file path, permissions, block information, etc. of all files and directories in the system. The FSImage file is parsed and its content is converted into a structured data format to facilitate subsequent data processing and analysis. According to the requirements of the distributed data processing platform (such as Hadoop, Spark), the parsed metadata file is converted into a table structure format that conforms to the target platform for easy import and use.

[0066] Select the target table (such as HBase table or Hive table), and import the converted metadata file into the metadata repository of the distributed data processing platform according to the pre-set partitioning rules (such as by date, file type, directory structure, etc.). Execute the data import operation to import the structured metadata files into the target table in batches to ensure data consistency and integrity during the import process.

[0067] In one specific implementation, in certain specific scenarios (such as when the system data volume is very large), a custom script or program is used to directly extract and parse the FSImage file, and the parsed data is directly stored in JSON format. Subsequently, the JSON data is imported into HBase or other distributed databases through Spark or other distributed computing frameworks to ensure data reading and processing efficiency in high-concurrency scenarios.

[0068] By extracting metadata from the distributed file system and importing it into the metadata repository of the distributed data processing platform, the data structure and file information in the distributed system can be effectively and centrally managed and analyzed. This not only improves the manageability and traceability of data, but also provides basic support for subsequent data processing and optimization. Especially in large-scale data processing scenarios, efficient metadata management is achieved, the scalability and flexibility of the system are enhanced, and the complexity of data management and system maintenance costs are significantly reduced.

[0069] S20, analyzing file information of metadata in the metadata repository through an offline analysis task of a distributed data processing platform, identifying cold data, and storing file path information of the cold data in a database;

[0070] In this embodiment, an offline analysis task is defined on a distributed data processing platform (such as Hadoop or Spark), and a batch processing job is used to periodically scan and analyze file information in a metadata repository.

[0071] The analyzed file information includes file path, file size, last access time, last modification time, creation time, and access frequency.

[0072] Use a scheduler (such as Oozie or Airflow) to periodically trigger offline analysis tasks to automatically run the task at preset time intervals, ensuring that the analysis process has minimal impact on the real-time system load.

[0073] The extracted file information is classified and filtered according to a predetermined set of rules. The predetermined rules may include: the file access frequency is lower than a certain threshold; the file's last access time is earlier than a set time point threshold; the file size exceeds or falls below a certain preset standard.

[0074] According to the above rules, cold data is automatically identified, and the file path information of these cold data is extracted as the basis for subsequent data migration and management. The identified cold data file path information is organized into a structured data format (such as a table) and stored in a dedicated database (such as MySQL, HBase, etc.) to facilitate subsequent query, statistics and data migration operations. The consistency and integrity of the data are ensured during the storage process, and data loss can be prevented through redundancy or backup mechanisms when necessary.

[0075] In one specific implementation, Hadoop's MapReduce programming model is used to extract required file information from the metadata repository in HDFS in the Map phase, and perform analysis and cold data identification in the Reduce phase. To improve efficiency, preprocessing can be performed in the Map phase, such as filtering out files that are obviously not cold data (such as recently accessed files), to reduce the computational burden of the Reduce phase.

[0076] In the MapReduce task, not only the overall information of the file is analyzed, but also the analysis is performed based on each data block of the file. By identifying the cold data characteristics of certain blocks, only some cold data blocks may be migrated instead of the entire file, thereby saving storage space. When storing the identified cold data file path information in the HBase table, its storage structure is expanded and redundant fields are added, such as access frequency, file size, last access time, etc. This information can be further used in subsequent decision-making or migration strategies.

[0077] By performing offline analysis tasks on a distributed data processing platform, cold data in the system can be automatically identified and classified, and its file path information can be stored. This can not only significantly reduce the system's reliance on high-performance computing resources and reduce the waste of computing resources, but also provide an accurate data foundation for subsequent data migration and storage optimization. By regularly analyzing and identifying cold data, the system can more flexibly adjust storage strategies and optimize resource allocation, thereby improving the efficiency and performance of overall storage management.

[0078] S30, generating a data migration plan based on the identified cold data, wherein the data migration plan includes a migration target location, a migration priority ranking, and a migration schedule;

[0079] In this embodiment, the system automatically generates a data migration plan based on the cold data identified in the previous step. The plan specifies the migration target location, priority, and schedule of each cold data file to ensure efficient execution of the data migration operation.

[0080] Determine the target location for migration based on the characteristics and storage requirements of cold data. The target location can include remote storage devices, cloud storage services, archive storage systems, etc. The selection criteria include factors such as storage cost, data access frequency, storage capacity, and data security requirements.

[0081] The migration priority of each cold data file is calculated based on the attributes of the cold data, such as file size, last access time, access frequency, etc. Priority sorting is usually based on comprehensive considerations such as resource utilization, storage urgency, and system load to ensure that high-priority data is migrated first.

[0082] After determining the priority, arrange a suitable migration time window for each cold data file based on the system's operating status and resource load. The time schedule needs to take into account the system's off-peak hours, bandwidth utilization, and possible business interruption risks.

[0083] In one specific implementation, in a local data center, cold data can be migrated to different storage media (such as SSD, HDD, tape storage) according to storage requirements. The selection of the migration target location depends on the access requirements and cost-effectiveness of the data, such as high-value cold data is migrated to low-speed HDD, and low-value cold data is migrated to tape storage.

[0084] Priority sorting can be adjusted dynamically. For example, during peak data center load periods, the migration priority of non-critical cold data can be automatically lowered, and the migration of critical business cold data can be prioritized to ensure the overall performance and stability of the system.

[0085] By analyzing the historical load data of the data center, the migration time can be intelligently scheduled. For example, the cold data migration task can be scheduled at night or during non-working hours to minimize the interference with other tasks of the system.

[0086] In another specific implementation, in a cloud environment, priority can be assigned based on storage levels, for example, frequently accessed cold data is preferentially migrated to a higher level of cloud storage (such as standard storage), while infrequently accessed cold data is migrated to a lower level of storage (such as cold storage or archive storage).

[0087] By generating a data migration plan based on the identified cold data, data storage management can be effectively optimized. The migration plan clarifies the migration target location, priority sorting, and migration schedule of cold data, making the storage and management of cold data more systematic and refined. It can reduce unnecessary computing resource consumption, improve the utilization of storage space, and avoid the impact of frequent access to hot data on system performance, ultimately improving the operating efficiency of the entire system and the use efficiency of storage resources.

[0088] S40, according to the data migration plan, migrating the cold data from the original storage location to the target remote storage, and updating the table partition path in the metadata repository to point to the new storage location after the migration;

[0089] In this embodiment, according to the previously generated data migration plan, the system starts the data migration task to migrate the identified cold data from the original storage location to the specified target remote storage. This step ensures that the cold data is removed from the original storage location that occupies more computing resources. During the migration process, an appropriate transmission protocol (such as FTP, SFTP, HTTP, etc.) can be selected to ensure the security and stability of data transmission.

[0090] Cold data can be migrated to different target remote storages, based on storage costs, storage requirements, data access frequency, data security, and compliance requirements. The target remote storage can be cloud storage, archive storage devices, or offsite storage systems. During the data migration process, the system may transfer large files in blocks to ensure transmission efficiency. After the migration is completed, the system will perform a data integrity check to ensure that the data has not been damaged or lost during the transmission process.

[0091] After the data migration is completed, the system automatically updates the table partition path information in the metadata repository and points the table partition to the newly migrated storage location. This operation ensures that subsequent access to the data can be correctly located at the new storage location, thereby ensuring data consistency and accessibility.

[0092] When updating the table partition path, you can also update the metadata information related to the file, such as storage location, file status, etc., so that the system can use the latest metadata for future data retrieval, analysis, and management. This step can further ensure the accuracy and reliability of system data management.

[0093] In one specific implementation, in a local data center, cold data can be transmitted in segments through an internal network, using the high transmission rate of the internal network bandwidth to split large files into smaller data packets for transmission, thereby reducing network congestion during transmission and improving transmission efficiency.

[0094] During the data migration process, you can choose to optimize the local storage path based on file size, type, and usage frequency, and update the path information in the metadata repository to ensure efficient use of local data storage.

[0095] By migrating cold data from the original storage location to the target remote storage according to the data migration plan and updating the table partition path in the metadata repository in a timely manner, it is possible to ensure efficient and secure data migration while ensuring the correctness of data access during the migration process. The migrated cold data can be automatically located to the new storage location, which improves the utilization of system storage resources, reduces storage costs, enhances the scalability and flexibility of the system, reduces the coupling between computing resources and storage resources, and thus improves the efficiency and performance of the overall system.

[0096] S50, after completing the data migration, deleting the migrated cold data in the original storage location.

[0097] In this embodiment, after the cold data is successfully migrated to the target remote storage, the system confirms the integrity and consistency of the migrated data through a verification mechanism (such as a verification code or a file hash value) to ensure that the data is not damaged or lost during the migration process.

[0098] After the migration is completed and confirmed, the system deletes the migrated cold data from the original storage location. The deletion operation can use the standard deletion instructions of the file system or perform specific deletion operations (such as file erasure, marking as overwriteable, etc.) based on the characteristics of the storage medium.

[0099] The deletion process uses secure deletion technology to ensure that the migrated cold data cannot be recovered by unauthorized access. This may include multiple overwrites, file fragmentation cleanup and other measures, especially for scenarios containing sensitive or compliance data.

[0100] After the deletion operation is completed, the system will re-mark the released storage space as free space and update the space utilization of the storage system to ensure that new data can fully utilize the released storage resources.

[0101] In another specific implementation, in a cloud environment, the system uses the automated deletion mechanism provided by the cloud service (such as Amazon S3's lifecycle management) to regularly clean up the migrated cold data and automatically release the space in the original storage location. This approach reduces manual intervention and ensures that the deletion operation is performed according to the predetermined policy. After the data is deleted, the cloud service can retain the version or snapshot of the deleted data for a period of time as a means of restoring the backup. After the data is confirmed to be correct, these snapshots will also be automatically deleted to completely release the storage space.

[0102] By deleting the migrated cold data in the original storage location after completing the data migration, the storage space is effectively released and optimized. Not only does it ensure the security and integrity of the cold data during the migration process, but it also avoids the risk of data leakage through the secure deletion technology, meeting data security and compliance requirements. In addition, the released storage space can be reused, thereby improving the overall resource utilization of the storage system, reducing the storage cost of the system, and improving the operating efficiency of the system.

[0103] The present invention relates to a data management method based on cold data migration, which extracts metadata from a distributed file system and imports it into a metadata repository of a distributed data processing platform. Through an offline analysis task, the file information in the metadata repository is analyzed, the cold data is identified, and its file path information is stored in a database. Based on the identified cold data, a data migration plan including a migration target location, migration priority sorting, and migration schedule is generated. According to the plan, the cold data is migrated from the original storage location to the target remote storage, and the table partition path in the metadata repository is updated to point to the new storage location. After the migration is completed, the cold data in the original storage location is deleted. The present invention realizes the effective separation of cold data and computing resources, optimizes the utilization efficiency of storage resources, reduces the overall resource consumption of the system, and improves the scalability and performance of the system.

[0104] In one embodiment, in the above S20, analyzing the file information of the metadata in the metadata repository and identifying the cold data includes:

[0105] S201, analyzing metadata in the metadata repository to extract file information related to the file, the file information including file path, last access time, last modification time, file size and creation time;

[0106] S202, setting an access frequency threshold and a time point threshold for identifying cold data;

[0107] S203, counting the file access frequency based on the file information, comparing the file access frequency with an access frequency threshold, and marking the file as cold data if the file access frequency is lower than the access frequency threshold;

[0108] S204, comparing the last access time of the file with a time point threshold, and if the last access time of the file is earlier than the time point threshold, marking the file as cold data;

[0109] S205, classifying the files according to their sizes, and performing a comprehensive evaluation based on the access frequency and the last access time of the files. If the comprehensive evaluation criteria are met, the files are marked as cold data.

[0110] In this embodiment, metadata related to the file is extracted from the metadata repository, including the file path, last access time, last modification time, file size, and creation time. The system traverses the records in the metadata repository to obtain detailed information for each file, providing basic data for subsequent analysis and identification.

[0111] System administrators or automated systems set predefined rules for identifying cold data based on business needs. These rules include:

[0112] Access frequency rule: Calculate the access frequency based on the file access records.

[0113] Last access time rule: judge based on the difference between the last access time of the file and the current time.

[0114] File size rule: Determines whether to classify a file as cold data based on its storage size.

[0115] The system calculates the access frequency of each file based on the extracted access record data. The calculated access frequency value is compared with the preset frequency threshold. If the frequency is lower than the set threshold, the file is marked as cold data. This step usually involves the application of statistical analysis tools or algorithms to ensure the accuracy of frequency calculation.

[0116] The system compares the last access time of the file with the preset time point threshold. If the last access time of the file is earlier than the threshold, the system considers that the file has not been accessed for a long time and should also be marked as cold data. This step can be achieved through a simple timestamp comparison.

[0117] Based on the file size, the files are graded. Combined with the file access frequency and the last access time, the system makes a comprehensive judgment to determine whether the file meets the cold data standard.

[0118] The system first classifies and grades the files according to their size. File size classification is usually divided into several intervals, each of which corresponds to different storage strategies and judgment criteria. For example: small files (such as less than 1MB); medium files (between 1MB and 1GB); large files (between 1GB and 10GB); and very large files (over 10GB).

[0119] Files in each size range may have different cold data determination criteria. For example, if a large file is not frequently accessed, it may be more easily marked as cold data, while a small file requires more conditions to be marked as cold data. For files that have been graded, the system further analyzes the file's access frequency and last access time. For example: Access frequency: The system calculates the number of times each file is accessed within a certain period of time and compares it with the preset access frequency threshold. If the file's access frequency is lower than the threshold, it is marked as potential cold data. Last access time: The system compares the file's last access time with the current time. If the file was last accessed earlier than the set time point threshold, the file is further marked as cold data.

[0120] The system conducts a comprehensive analysis of the results of file size, access frequency, and last access time. The comprehensive analysis can be performed in the following ways: Weighting method: Set a weight for each indicator and calculate a weighted average score. For example, for large files, the weight of file size may be larger, while for small files, the weights of access frequency and last access time may be larger. Files with a weighted average score below a certain threshold will be marked as cold data. Logical combination method: Set a combination of logical conditions. For example, only files with a file size greater than 1GB, an access frequency lower than a certain threshold, and a last access time greater than a certain time will be marked as cold data. This method can ensure more accurate cold data determination.

[0121] After the above comprehensive judgment, the files that meet the cold data criteria will be marked as cold data. If a file meets any rule or meets the conditions of cold data after comprehensive evaluation, it will be marked as cold data.

[0122] All files that meet the pre-defined rules or are comprehensively judged as cold data are marked. The marking information usually includes the file path and identification basis to facilitate subsequent data migration and management.

[0123] In one specific implementation, the system can combine weighted summation and logical combination to make a comprehensive judgment. First, the weighted summation method is used to calculate the comprehensive score, and then a secondary filtering is performed according to the logical conditions to ensure the accuracy of the judgment result. At the same time, the system can establish a rule library and dynamically update the judgment rules according to the actual operation situation. For example, in different time periods or under different storage pressures, the system will automatically adjust the judgment criteria to ensure the timeliness and accuracy of cold data identification.

[0124] This embodiment can efficiently and accurately identify and classify cold data by analyzing file information in the metadata repository and identifying cold data based on predetermined rules. This can not only optimize the use of system resources and reduce unnecessary storage and computing overhead, but also provide an accurate basis for subsequent data migration and management.

[0125] In one embodiment, the above S10 includes:

[0126] S101, exporting a current FSImage file from a distributed file system, wherein the FSImage file contains metadata of all files and directories in the file system;

[0127] S102, converting the exported FSImage file into a file format that complies with the table structure specification of the distributed data processing platform, wherein the conversion includes converting the metadata into a structured table format;

[0128] S103, selecting a target table in the distributed data processing platform, and importing the converted metadata file into a partition of the target table according to a specified partition rule, wherein the partition rule is based on date or file type;

[0129] S104, storing the imported metadata file in a metadata repository of the distributed data processing platform for use in data analysis tasks.

[0130] In this embodiment, the system exports the current FSImage file from a distributed file system (such as HDFS). The FSImage file is a metadata snapshot of the Hadoop distributed file system, which contains metadata information of all files and directories in the entire file system. This information includes the file path, size, block information, permissions, modification time, etc.

[0131] The FSImage file is a binary format file that is generated periodically to back up and restore the metadata of the file system. The system obtains the latest FSImage file through command line tools (such as Hadoop's hdfs dfsadmin-fetchImage) or API interfaces.

[0132] The exported FSImage file needs to be converted to the table structure format of the distributed data processing platform (such as Hive or HBase). This conversion process parses the binary data in the FSImage into readable structured data and converts it into a table format that conforms to the target platform (such as CSV, JSON, Parquet, etc.).

[0133] The conversion process includes mapping metadata entries to columns in the table, such as mapping file paths to path columns, mapping file sizes to size columns, etc. At the same time, the system may normalize the data as needed to make it suitable for storage and subsequent analysis.

[0134] The converted metadata file needs to be imported into the target table in the distributed data processing platform. The system selects the appropriate target table based on the characteristics of the data and analysis requirements.

[0135] During the import process, the system divides the metadata file into multiple partitions according to the specified partitioning rules and imports them into the table. The partitioning rules can be based on the file date, file type, or directory structure, etc., to optimize the performance of query and analysis. For example, file data can be imported into different partitions by date so that data for a specific time period can be quickly located during subsequent analysis.

[0136] The metadata file after importing into the target table is finally stored in the metadata repository of the distributed data processing platform for use in subsequent data analysis tasks. The metadata repository can be a database dedicated to storing and managing metadata (such as Hive Metastore or HBase table), which provides efficient metadata access and query support for data processing tasks.

[0137] This embodiment realizes the centralized management and efficient analysis of metadata by extracting metadata from the distributed file system and importing it into the metadata repository of the distributed data processing platform. It can convert all file and directory information in the file system into structured data, which is convenient for fast query and analysis operations in the distributed data processing platform. By importing metadata files according to specific rules, the system improves the organization and access efficiency of data storage, and provides a reliable data foundation for subsequent data processing and optimization. At the same time, the automated processing and storage mechanism of metadata significantly reduces the complexity of manual operations and improves the overall management efficiency of the system.

[0138] In one embodiment, in the above S30, generating a data migration plan based on the identified cold data includes:

[0139] S301, determining a migration target location of cold data, where the migration target location includes a remote storage, a cloud storage, or an archive storage device;

[0140] S302, calculating the migration priority of cold data based on file size, last access time and access frequency;

[0141] S303, dividing the cold data into multiple migration batches according to the migration priority, and formulating a migration schedule for each migration batch;

[0142] S304: Generate a data migration plan based on the migration target location, migration priority, and migration schedule of the cold data.

[0143] In this embodiment, the system determines the migration target location of each cold data according to the storage requirements and business policies of the cold data. The target location may include remote storage (such as an offsite data center), cloud storage (such as Amazon S3, Google Cloud Storage) or archival storage devices (such as tape libraries, optical disk storage). The basis for selection may include storage costs, data access requirements, data retention policies, and data security requirements.

[0144] The system calculates the migration priority of each cold data by analyzing the file size, last access time, and access frequency of the cold data. Priority calculation is usually based on the set weight. For example:

[0145] File size: Larger files may be given a higher migration priority because they take up more storage space.

[0146] Last access time: Files with older last access times may be given higher priority, indicating that they are unlikely to be accessed in the near future.

[0147] Access frequency: Files with low access frequency are migrated first because they have less impact on system usage.

[0148] The system divides cold data into multiple migration batches based on the calculated migration priority. Each batch contains a certain number of cold data files, and the system sets a corresponding migration schedule for each batch. The migration schedule usually takes into account system load, network bandwidth, and non-peak business time periods to ensure that the migration process does not have a significant impact on normal business operations.

[0149] The system generates a complete data migration plan based on the migration target location, migration priority and migration schedule of cold data. The plan includes the migration path, priority, target storage location and specific migration time point of each cold data file, and provides a detailed execution guide for the migration task.

[0150] In one specific implementation, in a cloud computing environment, the system can choose to migrate cold data to cloud storage in different geographic regions based on the storage cost, access latency, and compliance requirements of the data. For example, for non-critical cold data, the system can choose to store it in a region with lower storage costs.

[0151] In this embodiment, by generating a data migration plan based on the identified cold data, the system can achieve effective management of cold data. By determining the migration target location (such as remote storage, cloud storage or archival storage device) and calculating the migration priority of cold data based on file size, last access time and access frequency, the system can reasonably arrange the migration order and time of data. It not only optimizes the use of storage resources, but also effectively reduces storage costs and improves the efficiency of data storage and access. In addition, through the formulation of a detailed data migration plan, the orderliness and security of the data migration process are ensured, and the problems of data loss or system performance degradation that may occur during the migration process are avoided.

[0152] In one embodiment, the above S301 includes:

[0153] S3011, setting weights of file size, last access time, and access frequency according to the amount of storage resources occupied by the file size, the reflection of the last access time on the frequency of data use, and the data use reflected by the access frequency;

[0154] S3012, standardizing the data of file size, last access time and access frequency, and converting data of different scales into a unified numerical range;

[0155] S3013, based on the standardized file size, last access time and access frequency data, according to the set weight, calculate the migration priority score of each cold data file, the priority score is obtained by weighted summing the file size, last access time and access frequency according to their respective weights;

[0156] S3014, sort the cold data files according to the priority scores, and assign corresponding migration sequences to the cold data files.

[0157] In this embodiment, the system first analyzes the three key indicators of file size, last access time and access frequency to determine their importance in the migration priority calculation. This step is the basis for weight setting. The weight setting rules usually follow the following principles:

[0158] File size weight: File size reflects how much storage resources a file occupies. Larger files take up more storage space, so when the system needs to release storage resources, larger files are usually given a higher weight. This means that in the priority calculation, cold data with larger file sizes may be migrated first.

[0159] Weight of last access time: Last access time refers to the time when a file was last accessed. Files that have not been accessed for a long time are usually considered cold data because they are less relevant to current business operations or user needs. Therefore, the weight of last access time can be higher to ensure that files that have not been accessed for a long time are prioritized during the migration process.

[0160] Access frequency weight: Access frequency indicates the number of times a file is accessed within a certain period of time. Files with low access frequency may also be considered cold data. The system sets weights for access frequency based on historical access records to ensure that files with low access frequency receive appropriate migration priority.

[0161] When setting weights, the system usually seeks a balance between releasing storage resources and maintaining data activity. For example, if the current storage space is very tight, the system may increase the weight of file size; while when focusing on data availability, the system may increase the weight of access frequency and last access time.

[0162] The system can dynamically adjust the weight of each indicator based on the actual storage resource usage and business needs. For example, when the system detects that the storage space is about to run out, it can temporarily increase the weight of the file size to free up space as soon as possible; during the system idle period, the system may reduce the weight of the file size to prioritize the migration of files that have not been accessed for a long time.

[0163] In order to ensure that indicators of different scales can be reasonably weighted and summed, the system needs to standardize the data of file size, last access time, and access frequency. The purpose of standardization is to convert these data of different units and ranges into a unified numerical range (such as between 0 and 1), thereby eliminating the difference in data magnitude and making them comparable when weighted and summed. Standardization methods include but are not limited to:

[0164] Minimum-maximum normalization: The value of each indicator is scaled to between 0 and 1 by subtracting the minimum value and dividing it by the difference between the maximum and minimum values.

[0165] Z-score standardization: Subtract the mean from the value of each indicator and divide it by the standard deviation so that the data distribution conforms to the standard normal distribution (mean is 0 and standard deviation is 1).

[0166] Through normalization, the system ensures that the numerical comparison of file size, last access time and access frequency is meaningful, making the priority score obtained by weighted summation more accurate.

[0167] After completing the data normalization process, the system calculates the migration priority score for each cold data file based on the preset weight. This process includes the following steps:

[0168] Weighted summation: The standardized file size, last access time, and access frequency are multiplied by the corresponding weights to obtain the weighted value of each indicator. Then, these weighted values ​​are added together to obtain the comprehensive priority score of the file. The formula is as follows:

[0169] Priority score = W size × S size + W time × S time + W frequency × S frequency.

[0170] Among them, W represents the weight and S represents the standardized value.

[0171] The priority score is usually a value between 0 and 1. The lower the value, the more likely the file is to be cold data and should be migrated first. The score directly determines the order of data migration.

[0172] After obtaining the priority scores of all cold data files, the system sorts these files from low to high scores. The files with the lowest scores are considered to be the most consistent with the cold data standards and are therefore given priority for migration. After the sorting is completed, the system assigns a migration order to each file based on the sorting results to ensure that when the data migration task is actually executed, the files that need to be migrated most are given priority.

[0173] The system will execute the migration tasks of each cold data file in sequence according to the preset migration time window and resource availability, ensuring that the data can be transferred to the designated storage location safely and efficiently as planned.

[0174] In this embodiment, by calculating the migration priority of cold data based on file size, last access time and access frequency, the system can effectively identify the cold data that should be migrated first among many data files. By setting reasonable weights and standardization processing, the system can comprehensively consider factors such as storage resource utilization, data activity and system performance to ensure efficient and orderly execution of data migration plans.

[0175] In one embodiment, in the above S60, after deleting the migrated cold data in the original storage location, the process further includes:

[0176] S701, monitoring the access status of the migrated cold data, and recording the access time and frequency of the cold data files;

[0177] S702, judging whether the cold data becomes active again according to a preset activity threshold;

[0178] S703, if the access frequency of cold data reaches the set active threshold, a data migration plan is generated;

[0179] S704, executing the data migration plan to migrate the reactivated cold data to the original storage location or high-frequency storage area;

[0180] S705, updating the file path information in the metadata repository, and updating the new storage location of the migrated file to the corresponding table partition record;

[0181] S706, after completing the data migration, update the cache content and record the specific information of the migration operation.

[0182] In this embodiment, after the cold data is migrated to the target storage location (such as remote storage or archive storage), the system continues to monitor the data. The system records the access time and access frequency of each cold data file, which are used to determine the active state of the cold data.

[0183] The monitoring system is usually executed in real time or periodically to ensure that the situation of cold data being accessed again can be captured in time. The system records each access event through access logs or access counters, and updates the access frequency and last access time of cold data.

[0184] The system will determine whether the cold data has become active again based on the preset activity threshold. The activity threshold can include multiple indicators such as access frequency and last access time. For example, if the cold data is frequently accessed in a short period of time and the access frequency reaches or exceeds the preset threshold, the system will determine that the cold data has become active again.

[0185] The system's activity judgment criteria are usually based on business needs and historical data analysis, and can be adjusted according to specific usage scenarios. For example, some businesses may require greater sensitivity to file access frequency, while others may focus more on the last access time.

[0186] When the system determines that cold data has become active again, the system will automatically generate a data migration plan. The migration plan includes migrating cold data from the current storage location back to the original storage location or other high-frequency storage areas to more efficiently support subsequent access needs.

[0187] The migration plan will define the migration target location, migration priority, and specific migration schedule in detail. The system will determine the priority files to be migrated first based on the importance and activity of cold data to meet the real-time needs of the business.

[0188] The system executes the data migration task according to the data migration plan. The reactivated cold data will be migrated to the original storage location or other high-frequency storage areas to ensure that the data can quickly respond to user access requests.

[0189] During the data migration process, the system may compress or transfer the files in blocks to improve the migration efficiency and reduce the network bandwidth usage. At the same time, the system will monitor the transfer progress and data integrity during the migration process.

[0190] After the data is migrated back, the system needs to update the file path information in the metadata repository. The migrated files will be relocated to the new storage location, and the system needs to reflect this change in the metadata repository so that the data processing platform can correctly locate the files. Specifically, the system will update the table partition record in the metadata repository to point to the new storage location after the migration. This step ensures the accuracy of the data access path and the consistency of the system metadata.

[0191] After the data migration is completed, the system will update the relevant cache content to ensure that the file path information in the cache is consistent with the latest metadata, avoiding access problems caused by cache expiration or errors.

[0192] In addition, the system will record detailed information about the data migration operation, including migration time, migration target location, migration file list, migration process logs, etc. These records provide a basis for subsequent auditing, analysis, and troubleshooting.

[0193] This embodiment continues to monitor the access status of cold data after migration, so that the system can promptly determine whether the data has become active again, and automatically generate and execute a data migration plan when necessary. This ensures that the data can dynamically adjust the storage location according to actual access needs, optimizes the use of storage resources, and improves data accessibility and system response speed. By updating metadata and cache content, the system ensures the accuracy of the data access path and avoids access failures or performance issues caused by path errors.

[0194] In one embodiment, in the above S50, migrating the cold data from the original storage location to the target remote storage according to the data migration plan includes:

[0195] S501, reading the data migration plan to obtain a cold data file list, a migration target location, a migration priority, and a migration schedule;

[0196] S502, checking the availability of the migration target location, verifying whether the storage space meets the migration requirements, and verifying the stability of the network connection;

[0197] S503, starting migration tasks of cold data files in sequence according to the migration priority ranking and migration schedule in the data migration plan;

[0198] S504, during the migration process, split the cold data files that exceed the preset size threshold and transfer them part by part;

[0199] S505, real-time monitoring of transmission progress, transmission rate and stability of network connection, and recording of transmission data and status;

[0200] S506, after completing the data migration, perform data verification to check whether the data is intact during the transmission process, and record the log information of the migration process in detail.

[0201] In this embodiment, the system first reads the pre-generated data migration plan to obtain the cold data file list, migration target location, migration priority, and migration schedule. This step ensures that the system can accurately identify the files to be migrated and their corresponding target storage locations, and at the same time clearly define the priority and time window of the migration task.

[0202] Before starting the migration task, the system needs to verify the availability of the target storage location. This includes checking whether the target storage location has enough storage space to accommodate the cold data files to be migrated, and whether the network connection is stable and can support large-scale data transfer. The purpose of this step is to ensure that the migration process can proceed smoothly and avoid migration failures caused by insufficient storage space or unstable network.

[0203] The system starts the migration tasks of cold data files in sequence according to the migration priority and migration schedule in the data migration plan. Files with higher priority will be migrated first to ensure that important data is processed in a timely manner. The system starts the migration in an appropriate time window according to the plan to minimize interference with other system operations.

[0204] For cold data files that exceed the preset size threshold, the system will split them and then transfer them part by part. This method helps to improve transmission efficiency, especially when the network bandwidth is limited. Block transmission can reduce the amount of data transmitted in a single time, reduce network load, and reduce the risk of transmission failure.

[0205] The specific method of file segmentation may include cutting by fixed size or logical segmentation based on the structure of the file. The segmented file parts will be reassembled at the target storage location to ensure data integrity.

[0206] During the migration process, the system will monitor the progress of data transmission, transmission rate and stability of network connection in real time. The purpose of monitoring is to ensure the smooth progress of the transmission process, timely detect and handle possible abnormal situations (such as network jitter or transmission speed drop), so as to ensure the quality of data transmission.

[0207] The system will record the data status and transfer log during the transfer process so that problems can be quickly located and resolved. After the data transfer is completed, the system will verify the migrated data to ensure that the data has not been damaged or lost during the transfer process. Common data verification methods include checksum, hash value comparison (such as MD5 or SHA-256), etc.

[0208] After completing the data verification, the system will record the log information of the entire migration process in detail, including migration time, transmission rate, data integrity verification results, etc. The log information provides an important basis for subsequent auditing, analysis and troubleshooting.

[0209] In this embodiment, the system achieves efficient and reliable cold data migration by migrating cold data from the original storage location to the target remote storage according to the data migration plan. By checking the availability of the target storage location and the stability of the network connection, the system ensures the smooth progress of the migration process. By splitting large files and transmitting them part by part, the system optimizes the utilization of network bandwidth and reduces the risk of transmission failure. Real-time monitoring and data verification further ensure the integrity and security of data during transmission. Finally, detailed migration log records provide strong support for system auditing, analysis and troubleshooting, improving system management efficiency and data security.

[0210] The present invention also provides a data management device based on cold data migration, referring to Figure 2 , Figure 2 This is a functional module diagram of a preferred embodiment of a data management device based on cold data migration of the present invention. The data management device based on cold data migration includes:

[0211] The metadata extraction and import module 10 is used to extract metadata from the distributed file system and import the metadata into the metadata repository of the distributed data processing platform;

[0212] The cold data identification and storage module 20 is used to analyze the file information of the metadata in the metadata repository through the offline analysis task of the distributed data processing platform, identify the cold data, and store the file path information of the cold data in the database;

[0213] A data migration plan generating module 30, for generating a data migration plan based on the identified cold data, wherein the data migration plan includes a migration target location, a migration priority ranking, and a migration schedule;

[0214] A data migration and metadata update module 40, configured to migrate the cold data from the original storage location to the target remote storage according to the data migration plan, and update the table partition path in the metadata repository to point to the new storage location after the migration;

[0215] The data cleaning module 50 is used to delete the migrated cold data in the original storage location after completing the data migration.

[0216] The specific implementation of the data management device based on cold data migration of the present invention is basically the same as the various embodiments of the data management method based on cold data migration described above, and will not be described in detail herein.

[0217] The present invention also provides a data management device based on cold data migration, such as Figure 3 As shown, the data management device based on cold data migration may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0218] Those skilled in the art will understand that Figure 3 The hardware structure of the data management device based on cold data migration shown in the figure does not constitute a limitation on the data management device based on cold data migration, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0219] like Figure 3As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a data management program based on cold data migration. The operating system is a program for managing and controlling data management devices and software resources based on cold data migration, and supports the operation of the network communication module, the user interface module, the data management program based on cold data migration, and other programs or software; the network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.

[0220] exist Figure 3 In the hardware structure of the data management device based on cold data migration shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the data management program based on cold data migration stored in the memory 1005, and perform the same operations as the data management method based on cold data migration.

[0221] The specific implementation of the data management device based on cold data migration of the present invention is basically the same as the various embodiments of the data management method based on cold data migration described above, and will not be described in detail herein.

[0222] In addition, an embodiment of the present invention further proposes a computer storage medium, on which a data management program based on cold data migration is stored. When the data management program based on cold data migration is executed by a processor, the steps of the data management method based on cold data migration as described above are implemented.

[0223] The specific implementation of the computer storage medium of the present invention is basically the same as the above-mentioned data management method based on cold data migration, and will not be described in detail here.

[0224] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A data management method based on cold data migration, characterized in that: The following steps are involved: Extracting metadata from a distributed file system, and importing the metadata into a metadata repository of a distributed data processing platform; By means of an offline analysis task of a distributed data processing platform, file information of metadata in the metadata repository is analyzed, cold data is identified, and file path information of the cold data is stored in a database; Based on the identified cold data, generate a data migration plan, the data migration plan including a migration target location, a migration priority ranking, and a migration schedule; According to the data migration plan, migrate the cold data from the original storage location to the target remote storage, and update the table partition path in the metadata repository to point to the new storage location after the migration; After completing the data migration, delete the migrated cold data in the original storage location.

2. The data management method based on cold data migration according to claim 1, characterized in that: Analyzing file information of metadata in the metadata repository and identifying cold data, including: Analyzing the metadata in the metadata repository to extract file information related to the file, the file information including file path, last access time, last modification time, file size and creation time; Set access frequency thresholds and time point thresholds for identifying cold data; Based on the file information, count the file access frequency, compare the file access frequency with the access frequency threshold, and if the file access frequency is lower than the access frequency threshold, mark the file as cold data; Compare the last access time of the file with a time point threshold, and if the last access time of the file is earlier than the time point threshold, mark the file as cold data; Files are graded according to their size and comprehensively evaluated based on the access frequency and last access time of the files. If the comprehensive evaluation criteria are met, the file is marked as cold data.

3. The data management method based on cold data migration according to claim 1, characterized in that: Extracting metadata from a distributed file system and importing the metadata into a metadata repository of a distributed data processing platform includes: Export the current FSImage file from the distributed file system, wherein the FSImage file contains metadata of all files and directories in the file system; Convert the exported FSImage file into a file format that complies with the table structure specification of the distributed data processing platform, wherein the conversion includes converting the metadata into a structured table format; Selecting a target table in a distributed data processing platform, and importing the converted metadata file into a partition of the target table according to a specified partitioning rule, wherein the partitioning rule is based on a date or a file type; The imported metadata files are stored in the metadata repository of the distributed data processing platform for use in data analysis tasks.

4. The data management method based on cold data migration according to claim 1, characterized in that: Based on the identified cold data, generate a data migration plan, including: Determine a migration target location for cold data, wherein the migration target location includes remote storage, cloud storage, or an archive storage device; Calculate the migration priority of cold data based on file size, last access time and access frequency; Divide cold data into multiple migration batches according to migration priority, and set a migration schedule for each migration batch; Generate a data migration plan based on the migration target location, migration priority, and migration schedule of the cold data.

5. The data management method based on cold data migration as claimed in claim 4, characterized in that: Calculate the migration priority of cold data based on file size, last access time, and access frequency, including: The weights of file size, last access time and access frequency are set according to the amount of storage resources occupied by file size, the reflection of last access time on data usage frequency and the data usage reflected by access frequency; Standardize the data of file size, last access time and access frequency, and convert data of different scales into a unified numerical range; Based on the standardized file size, last access time, and access frequency data, the migration priority score of each cold data file is calculated according to the set weight. The priority score is obtained by weighting the file size, last access time, and access frequency according to their respective weights. Sort the cold data files according to their priority scores and assign corresponding migration orders to the cold data files.

6. The data management method based on cold data migration according to claim 1, characterized in that: After deleting the migrated cold data in the original storage location, also include: Monitor the access to migrated cold data and record the access time and frequency of cold data files; Determine whether cold data becomes active again based on the preset activity threshold; If the access frequency of cold data reaches the set active threshold, a data migration plan is generated; Execute the data migration plan to migrate the reactivated cold data to the original storage location or high-frequency storage area; Update the file path information in the metadata repository and update the new storage location of the migrated file to the corresponding table partition record; After the data is migrated back, the cache content is updated and the specific information of the migration operation is recorded.

7. The data management method based on cold data migration according to claim 1, characterized in that: Migrating the cold data from the original storage location to the target remote storage according to the data migration plan includes: Read the data migration plan to obtain a cold data file list, a migration target location, a migration priority, and a migration schedule; Check the availability of the migration target location, verify whether the storage space meets the migration requirements, and verify the stability of the network connection; Start the migration tasks of cold data files in sequence according to the migration priority and migration schedule in the data migration plan; During the migration process, cold data files that exceed the preset size threshold are split and transferred part by part; Monitor the transmission progress, transmission rate and stability of network connection in real time, and record the transmission data and status; After completing the data migration, perform data verification to check whether the data is intact during the transmission process, and record the log information of the migration process in detail.

8. A data management device based on cold data migration, characterized in that: The data management device based on cold data migration includes: A metadata extraction and import module, used to extract metadata from a distributed file system and import the metadata into a metadata repository of a distributed data processing platform; A cold data identification and storage module is used to analyze the file information of the metadata in the metadata repository through an offline analysis task of a distributed data processing platform, identify cold data, and store the file path information of the cold data in a database; A data migration plan generation module, used to generate a data migration plan based on the identified cold data, wherein the data migration plan includes a migration target location, a migration priority ranking, and a migration schedule; A data migration and metadata update module, used to migrate the cold data from the original storage location to the target remote storage according to the data migration plan, and update the table partition path in the metadata repository to point to the new storage location after the migration; The data cleaning module is used to delete the migrated cold data in the original storage location after completing the data migration.

9. A data management device based on cold data migration, characterized in that: The data management device based on cold data migration includes a memory, a processor, and a data management program based on cold data migration stored in the memory and executable on the processor. When the data management program based on cold data migration is executed by the processor, the steps of the data management method based on cold data migration as described in any one of claims 1 to 7 are implemented.

10. A computer storage medium, characterized in that: The storage medium stores a data management program based on cold data migration, and when the data management program based on cold data migration is executed by the processor, the steps of the data management method based on cold data migration according to any one of claims 1 to 7 are implemented.

Images