Archiving management method and system for massive small files

Abstract

The present invention provides an archiving management method and system for a large number of small files, belonging to the field of data management. The method comprises: real-time monitoring of the I / O activities of a computer disk, storing the identified changed files in a transfer station composed of an SSD array, generating a metadata index and an operation log and performing regular analysis, determining the retention period and archiving strategy of the files, and regularly creating a copy of the files in the SSD array in a virtual machine environment, comparing the copy with the archived data in a long-term storage, identifying newly added, modified or deleted files, marking and transferring them to the long-term storage for archiving, and creating independent archiving records with timestamps; based on the metadata index, the operation log and the independent archiving record, calling the data in the long-term storage to retrieve and use the archived files. The present invention realizes efficient archiving and management of a large number of small files, and improves the retrievability and accessibility of data.

Description

Technical Field

[0001] The present invention belongs to the field of data management, and in particular relates to an archiving management method and system for massive small files. Background Art

[0002] With the popularization of computers and networks, the data that needs to be managed is also growing exponentially. Among them, unstructured data is characterized by small files, fragmentation, difficulty in collection and management. Faced with massive data consisting of a large number of KB-level small files scattered in a complex directory structure, how to effectively obtain and manage massive small files has become a huge challenge.

[0003] In the prior art, the traditional archiving method relies on the file system to access and manage files; however, when processing files, the file system needs to perform operations such as opening, reading, and closing each file, and identify the data to be obtained through data comparison, which consumes a lot of time and resources for file opening, closing, comparison, and other operations, and has low data processing efficiency and is unable to cope with a large number of small and fragmented files, resulting in performance bottlenecks when obtaining and archiving a large number of small files. At the same time, when the traditional method manages the number of archives, it is usually necessary to copy the data from the archive storage to the production environment and convert it into a format that can be recognized by the application before management. The copying and conversion process is not only time-consuming and inefficient, but even if the archive is completed, the retrievability and accessibility of the large number of small files are not high, and the actual value of the archived data cannot be fully utilized, and the availability is not high. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, the present invention aims to provide an archiving management method and system for massive small files, configure a solid-state disk array, monitor disk I / O activities in real time, autonomously monitor files by capturing data changes, effectively organize, transmit and record each piece of data, thereby realizing the archiving and management of massive small files, improving the processing efficiency of small file data, optimizing the archiving process, and improving the retrievability and accessibility of small file data, thereby giving full play to the value of archived data.

[0005] In order to achieve the above purpose, the embodiment of the present invention adopts the following technical solution:

[0006] In a first aspect, an embodiment of the present invention provides an archiving management method for a large number of small files, comprising the following steps:

[0007] Step S1, real-time monitoring of the I / O activity of the computer disk that generates a large number of small files at the physical level, identifying and recording each change of the computer disk;

[0008] Step S2, storing the changed files in a transfer station composed of an SSD array, and generating a metadata index and an operation log for each changed file;

[0009] Step S3, regularly analyzing the metadata index and operation log in the SSD array, automatically identifying the importance and priority of files, and determining the retention period and archiving strategy of files;

[0010] Step S4, according to the determined file retention period and archiving policy, regularly create a complete copy of the temporary file currently stored in the SSD array in the virtual machine environment to provide a benchmark for the subsequent data comparison;

[0011] Step S5, in the virtual machine environment, comparing the copy with the archived data in the long-term storage, and identifying in the comparison process the files newly added, modified or deleted in the copy relative to the archived data;

[0012] Step S6, marking the newly added or modified file and transferring it to a long-term storage for archiving, and creating an independent archiving record with a timestamp for the archived file at the current time point;

[0013] Step S7, based on the metadata index, operation log and independent archive record, call the data in the long-term storage to retrieve and use the archived file.

[0014] As a preferred embodiment of the present invention, in step S1, disk I / O activity monitoring adopts disk drive level monitoring to detect all disk I / O operations in real time, including not only changes in file content but also adjustments to file attributes and structures.

[0015] As a preferred embodiment of the present invention, in step S1, the SSD array adopts a management architecture of RAID configuration.

[0016] As a preferred embodiment of the present invention, the metadata index includes basic attributes and complex attributes of the file; the operation log includes the data change history, transmission and archiving process.

[0017] As a preferred embodiment of the present invention, the basic attributes include file name, size and modification time.

[0018] As a preferred embodiment of the present invention, the complex attributes include access rights, version history and association relationships.

[0019] As a preferred embodiment of the present invention, in step S2, the changed files are stored in a transfer station composed of an SSD array. The specific process includes:

[0020] Step S21, marking the changed files as to be transmitted;

[0021] Step S22, packaging and marking the data to be transmitted to form a data packet, and assigning a unique identifier to each data packet, wherein the unique identifier includes information about the data change type, timestamp, and related metadata, so as to facilitate accurate transmission, archiving, and retrieval;

[0022] Step S23, compressing the data packets to reduce the volume of the transmitted data, integrating the compressed data packets in a batch processing manner, transmitting and storing them to the SSD array transfer station, completing the transmission task while reducing the network transmission overhead.

[0023] As a preferred embodiment of the present invention, step S7 retrieves and uses the archived file, specifically comprising the following steps:

[0024] Step S71, the user inputs access conditions through a graphical interface;

[0025] Step S72, querying according to the access conditions, quickly locating archived data based on metadata indexes, operation logs and / or independent archive records, and finding archived files that meet the conditions in the long-term storage;

[0026] Step S73, after retrieving the required archived files, the user directly accesses or restores the required files to the production environment.

[0027] As a preferred embodiment of the present invention, the access condition includes a timestamp, a file type, and / or a keyword.

[0028] In a second aspect, an embodiment of the present invention further provides an archiving management system for a large number of small files, the system comprising a disk activity monitoring module, an SSD array storage module, an index module, an archiving analysis module, a virtual machine environment module, and a long-term storage module; wherein:

[0029] The disk activity monitoring module is used to perform real-time physical-level monitoring of the I / O activity of the computer disk that generates a large number of small files, identify and record each change of the computer disk, and send the changed files to the SSD array storage module and the index module;

[0030] The SSD array storage module is used as a transfer station to store changed files;

[0031] The index module is used to generate a metadata index and an operation log for each changed file stored in the SSD array, to create an independent archive record with a timestamp for the archived file stored in the long-term storage, and to call the data in the long-term storage to retrieve and use the archived file according to the metadata index, the operation log and the independent archive record;

[0032] The archiving analysis module is used to regularly analyze the metadata index and operation log in the SSD array, automatically identify the importance and priority of files, and determine the retention period and archiving strategy of files, so as to achieve data life cycle management, ensure that key data is retained for a long time and obsolete data is processed in a timely manner;

[0033] The virtual machine environment module is used to periodically create a complete copy of the temporary files currently stored in the SSD array as a benchmark according to the determined file retention period and archiving strategy, compare the copy with the archived data in the long-term storage, and identify the newly added, modified or deleted files in the copy relative to the archived data during the comparison process; and mark the newly added or modified files and transfer them to the NAS for archiving;

[0034] The network attached storage device long-term storage module is used for long-term storage of archived files.

[0035] The technical solution provided by the embodiment of the present invention has the following beneficial effects:

[0036] The archiving management method and system for massive small files provided by the embodiment of the present invention improves the processing and synchronization efficiency of massive small files by configuring SSD arrays and real-time data monitoring mechanisms, and is particularly suitable for data-intensive applications; long-term data preservation is achieved through long-term storage to meet compliance and regulatory requirements, especially for industries that need to preserve important data for a long time; metadata indexes and operation logs are generated while monitoring files to provide a basis for file archiving, while providing fast and efficient data retrieval capabilities, making access to archived data more convenient. The present invention reduces storage costs and improves storage efficiency; simplifies data archiving and management work, and reduces human errors; at the same time, it can adapt to different hardware and network environments, reduces dependence on specific hardware, and improves the adaptability and flexibility of the system; real-time monitoring and instant processing mechanisms significantly reduce the risk of data loss, ensure data integrity and recoverability, adapt to the efficient processing and archiving of large-scale data, and ensure the long-term availability of the system.

[0037] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1It is a flowchart of the archiving management method for massive small files described in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations. It should be noted that the embodiments of the present invention and the features in the embodiments can also be combined with each other without conflict.

[0041] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In the description of the present invention, the terms "first", "second", "third", "fourth", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0042] The embodiment of the present invention provides an archiving management method and system for a large number of small files, targeting small, scattered, fragmented and unstructured files of huge number. The speed and efficiency of data processing are improved by configuring a solid state disk (SSD) array. This configuration not only accelerates data access and processing, but also provides strong support for real-time data processing. Through detailed disk I / O activity monitoring, accurate capture of data changes is achieved, laying a foundation for timely processing and archiving of data. For the corresponding files that have been monitored and need to be processed, each piece of data is organized, transmitted and recorded, and corresponding metadata indexes and operation logs are generated to achieve archiving, searching and using of files. Long-term storage devices such as network attached storage (NAS) are set to store data for a long time to ensure the authenticity and traceability of data. At the same time, through the data pull-up mechanism and refined data comparison process of simulating the production environment, the data to be archived is accurately identified, the integrity and accuracy of data archiving are guaranteed, and the safe, efficient and convenient management of data during the life cycle is ensured. The retrieval and access of data are achieved based on metadata indexes and operation logs, and there is no need to repeatedly copy and store data, so as to give full play to the value of data.

[0043] like Figure 1 As shown, the archiving management method for massive small files includes the following steps:

[0044] Step S1, real-time monitoring of the I / O activities of the computer disk that generates a large number of small files at the physical level, identifying and recording each change of the computer disk.

[0045] In this step, the I / O activity monitoring of the computer disk adopts the monitoring at the disk driver level to detect all I / O operations of the disk in real time, including not only changes in file content, but also adjustments to file attributes and structures, such as the addition, modification and deletion of files. Disk driver-level monitoring directly monitors disk I / O activities at the bottom layer of the operating system, focusing on activities at the physical disk level, reducing redundant operations in the processing process, and only focusing on those data areas that actually change, including all operations on data, such as writing, updating, deleting, etc., not only covering file-level activities, but also going deep into the disk driver level, accurately capturing every data change, ensuring that every data change can be identified immediately, thereby providing an accurate basis for subsequent data processing and archiving.

[0046] This step reduces the reliance on the upper file system and reduces the system load by directly monitoring disk I / O activities at the bottom layer, especially when dealing with high-density data changes. Real-time monitoring ensures that data changes are not missed and improves the accuracy of data synchronization and backup. Accurate and efficient data change monitoring supports more efficient data management strategies, making data archiving, retrieval and recovery processes faster and more reliable. Through this bottom-level monitoring and response mechanism, the data synchronization process becomes more efficient. The system only processes those data areas that have changed, rather than the entire file system, reducing unnecessary data processing and thus improving the overall data synchronization efficiency.

[0047] Step S2, storing the changed files in a transfer station composed of an SSD array, and generating a metadata index and an operation log for each changed file.

[0048] In this step, the configuration of the SSD array is crucial. The SSD array can significantly reduce data access latency based on its fast data read and write capabilities, and is particularly suitable for processing real-time changing data; as a transfer station, it temporarily stores data to be archived to ensure rapid accessibility of the data before archiving. By configuring the SSD array, a large number of small files can be written quickly, thereby achieving real-time monitoring of disk I / O activities. The SSD array uses an efficient connection and management architecture, such as RAID configuration, to enhance data read and write speeds and redundant protection, which not only improves overall storage performance, but also ensures data security and reliability.

[0049] The metadata index and operation log generated in this step are used for subsequent archiving of files on the one hand, and for retrieval and reference of archived files on the other hand. As the identity of the file, it plays an important role in the subsequent processing and application of the file. The metadata index includes basic attributes of the file such as name, size, modification time, etc., as well as more complex attributes such as access rights, version history and association relationships. The metadata index is designed to quickly locate and retrieve data and improve management efficiency. The operation log tracks the change history of data and records the data transmission and archiving process to facilitate auditing and backtracking.

[0050] In this step, the changed files are temporarily stored in the SSD array transfer station. The specific data transmission process is as follows:

[0051] Step S21, marking the changed files as to be transmitted;

[0052] Step S22, packaging and marking the data to be transmitted to form a data packet, and assigning a unique identifier to each data packet, wherein the unique identifier includes information about the data change type, timestamp, and related metadata, so as to facilitate accurate transmission, archiving, and retrieval;

[0053] Step S23, compressing the data packets to reduce the volume of the transmitted data, integrating the compressed data packets in a batch processing manner, transmitting and storing them to the SSD array transfer station, completing the transmission task while reducing the network transmission overhead.

[0054] Through the above data transmission process, by adopting data management and transmission strategies, the load of network transmission is effectively reduced and the response speed of data changes is accelerated. It is especially suitable for environments that need to process large amounts of dynamically changing data, thereby improving transmission efficiency. It ensures data consistency and accuracy. The refined data packaging and labeling process ensures the consistency and accuracy of data during transmission and archiving, which is beneficial to subsequent data analysis and use. It optimizes resource utilization. Through intelligent data management, the system can more effectively utilize network and storage resources, reducing the overall cost of data processing and improving the economy and environmental friendliness of the system.

[0055] Step S3, regularly analyze the metadata index and operation log in the SSD array, automatically identify the importance and priority of files, and determine the retention period and archiving strategy of files, so as to achieve data life cycle management, ensure that key data is retained for a long time and obsolete data is processed in a timely manner.

[0056] In this step, the archiving strategy includes: archiving newly added files or files that have been archived and changed; permanently storing the archived data, etc.

[0057] Step S4, based on the determined file retention period and archiving policy, regularly create a complete copy of the temporary files currently stored in the SSD array in the virtual machine environment to provide a benchmark for subsequent data comparison.

[0058] In this step, the virtual machine environment provides a copy and a comparison environment for data comparison, thereby providing a basis for data archiving.

[0059] Step S5, in a virtual machine environment, comparing the copy with the archived data in the long-term storage, and identifying in the comparison process the files that are newly added, modified or deleted in the copy relative to the archived data.

[0060] This step is the file comparison process. By simulating the data pulling mechanism of the production environment and the refined data comparison process, the data that needs to be archived can be accurately identified, the integrity and accuracy of data archiving can be guaranteed, and the safe, efficient and convenient management of data during its life cycle can be ensured.

[0061] Step S6, marking the newly added or modified file and transferring it to a long-term storage for archiving, while creating an independent archiving record with a timestamp for the archived file at the current time point to maintain data integrity and traceability.

[0062] In this step, long-term storage is used for data archiving, which can effectively perform version control and data backup. The long-term storage includes: Network Attached Storage (NAS), storage that supports NAS protocol, Blu-ray disc library, and / or shared storage. This is especially important for application scenarios that need to trace historical versions or restore old data.

[0063] The periodic migration of temporary files in the SSD array to the long-term storage specifically includes the following steps:

[0064] Through the above archiving process, the real-time consistency and integrity of archived data are ensured, thereby effectively protecting key business data. Accurate change tracking: The system's comparison strategy can accurately identify any changes in data, including new, modified or deleted content, to ensure the integrity and accuracy of archived records. Efficient long-term data preservation: Using long-term storage as an archiving medium can retain important data for a long time, and through a sophisticated version control mechanism, it supports accurate management and access to historical data.

[0065] Step S7, based on the metadata index, operation log and independent archive record, call the data in the long-term storage to retrieve and use the archived file.

[0066] In this step, users can quickly locate and access required archived files from long-term storage based on conditions such as date, metadata index, operation log and / or independent archive records, thereby improving the user's efficiency and experience in archived data management and use.

[0067] In this step, the retrieval and use of the archived files specifically includes the following steps:

[0068] Step S71: The user inputs access conditions through a graphical interface.

[0069] Step S72, query according to access conditions (such as timestamp, file type, keywords, etc.), quickly locate archived data based on metadata indexes, operation logs and / or independent archive records, and find archived files that meet the conditions in long-term storage.

[0070] Step S73, after retrieving the required archived files, the user directly accesses or restores the required files to the production environment.

[0071] Based on the same idea, an embodiment of the present invention also provides an archiving management system for massive small files, which includes a disk activity monitoring module, an SSD array storage module, an index module, an archiving analysis module, a virtual machine environment module and a long-term storage device.

[0072] The disk activity monitoring module is used to perform real-time physical-level monitoring of the I / O activity of the computer disk that generates a large number of small files, identify and record each change of the computer disk, and send the changed files to the SSD array storage module and the index module;

[0073] The SSD array storage module is used as a transfer station to store changed files;

[0074] The index module is used to generate a metadata index and an operation log for each changed file stored in the SSD array, to create an independent archive record with a timestamp for the archived file stored in the long-term storage, and to call the data in the long-term storage to retrieve and use the archived file according to the metadata index, the operation log and the independent archive record;

[0075] The archiving analysis module is used to regularly analyze the metadata index and operation log in the SSD array, automatically identify the importance and priority of files, and determine the retention period and archiving strategy of files, so as to achieve data life cycle management, ensure that key data is retained for a long time and obsolete data is processed in a timely manner;

[0076] The virtual machine environment module is used to periodically create a complete copy of the temporary files currently stored in the SSD array as a benchmark according to the determined file retention period and archiving strategy, compare the copy with the archived data in the long-term storage, and identify the newly added, modified or deleted files in the copy relative to the archived data during the comparison process; and mark the newly added or modified files and transfer them to the long-term storage for archiving;

[0077] The long-term storage is used for long-term storage of archived files.

[0078] In this embodiment, each module is implemented by a processor and / or a memory. The processor may be, but is not limited to, a microprocessor MPU, a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, etc. The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0079] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, computer or data center.

[0080] It should also be noted that the archiving management system for massive small files described in this embodiment corresponds to the archiving management method for massive small files. The description and limitation of the method are also applicable to the system and will not be repeated here.

[0081] It can be seen from the above technical solutions that the archiving management method and system for massive small files provided by the embodiment of the present invention improves the processing and synchronization efficiency of massive small files by configuring SSD arrays and real-time data monitoring mechanisms, and is particularly suitable for data-intensive applications; long-term data preservation is achieved through long-term storage to meet compliance and regulatory requirements, especially for industries that need to preserve important data for a long time; metadata indexes and operation logs are generated while monitoring files to provide a basis for file archiving, and fast and efficient data retrieval capabilities are provided, making access to archived data more convenient. The present invention reduces storage costs and improves storage efficiency; simplifies data archiving and management work, and reduces human errors; at the same time, it can adapt to different hardware and network environments, reduces dependence on specific hardware, and improves the adaptability and flexibility of the system; real-time monitoring and immediate processing mechanisms significantly reduce the risk of data loss, ensure data integrity and recoverability, adapt to the efficient processing and archiving of large-scale data, and ensure the long-term availability of the system.

[0082] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. It is not intended to limit the scope of the invention claimed for protection, but only represents the preferred embodiment of the present invention. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solution formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the inventive concept. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

Claims

1. A method for archiving and managing a large number of small files, characterized in that: The steps include: Step S1, real-time monitoring of the I / O activity of the computer disk that generates a large number of small files at the physical level, identifying and recording each change of the computer disk; Step S2, storing the changed files in a transfer station composed of a solid state disk SSD array, and generating a metadata index and an operation log for each changed file; the specific process of storing the changed files in the transfer station composed of the SSD array includes: Step S21, marking the changed files as to be transmitted; Step S22, packaging and marking the data to be transmitted to form a data packet, assigning a unique identifier to each data packet, wherein the unique identifier includes information about the data change type, timestamp, and related metadata, so as to facilitate accurate transmission, archiving, and retrieval; Step S23, compressing the data packets to reduce the volume of the transmitted data, integrating the compressed data packets in a batch processing manner, transmitting and storing them to the SSD array transfer station, completing the transmission task while reducing the network transmission overhead; Step S3, regularly analyzing the metadata index and operation log in the SSD array, automatically identifying the importance and priority of files, and determining the retention period and archiving strategy of files; Step S4, according to the determined file retention period and archiving policy, regularly create a complete copy of the temporary file currently stored in the SSD array in the virtual machine environment to provide a benchmark for the subsequent data comparison; Step S5, in the virtual machine environment, comparing the copy with the archived data in the long-term storage, and identifying in the comparison process the files newly added, modified or deleted in the copy relative to the archived data; Step S6, marking the newly added or modified file and transferring it to a long-term storage for archiving, and creating an independent archiving record with a timestamp for the archived file at the current time point; Step S7, based on the metadata index, operation log and independent archive record, call the data in the long-term storage to retrieve and use the archived file.

2. The archiving management method for massive small files according to claim 1 is characterized in that: In step S1, disk I / O activity monitoring uses disk driver-level monitoring to detect all disk I / O operations in real time, including changes in file content, adjustments to file attributes and structures.

3. The archiving management method for massive small files according to claim 1 is characterized in that: In step S1, the SSD array adopts a RAID configuration management architecture.

4. The archiving management method for massive small files according to claim 1 is characterized in that: The metadata index includes the basic attributes and complex attributes of the file; the operation log includes the data change history, transmission and archiving process.

5. The archiving management method for massive small files according to claim 4 is characterized in that: The basic attributes include the file name, size, modification time, and MD5 attribute information of the file.

6. The archiving management method for massive small files according to claim 4 is characterized in that: The complex attributes include access rights, version history, and association relationships.

7. The archiving management method for massive small files according to claim 1 is characterized in that: Step S7 retrieves and uses the archived files, specifically including the following steps: Step S71, the user inputs access conditions through a graphical interface; Step S72, querying according to the access conditions, quickly locating archived data based on metadata indexes, operation logs and / or independent archive records, and finding archived files that meet the conditions in the long-term storage; Step S73, after retrieving the required archived files, the user directly accesses or restores the required files to the production environment.

8. The archiving management method for massive small files according to claim 7 is characterized in that: The access condition includes a timestamp, a file type, and / or a keyword.

9. An archiving management system for massive small files, characterized in that: The system includes a disk activity monitoring module, an SSD array storage module, an index module, an archive analysis module, a virtual machine environment module, and a long-term storage; wherein, The disk activity monitoring module is used to perform real-time physical-level monitoring of the I / O activity of the computer disk that generates a large number of small files, identify and record each change of the computer disk, and send the changed files to the SSD array storage module and the index module; The SSD array storage module is used as a transfer station to store changed files; and is specifically used to mark changed files as to be transmitted, package and mark the data to be transmitted to form a data packet, and assign a unique identifier to each data packet, wherein the unique identifier includes information about the data change type, timestamp, and related metadata, so as to facilitate accurate transmission, archiving, and retrieval, and compress the data packet to reduce the volume of the transmitted data, and use a batch processing method to integrate the compressed data packets, transmit and store them to the SSD array transfer station, thereby completing the transmission task and reducing the network transmission overhead; The index module is used to generate a metadata index and an operation log for each changed file stored in the SSD array, to create an independent archive record with a timestamp for the archived file stored in the long-term storage, and to call the data in the long-term storage to retrieve and use the archived file according to the metadata index, the operation log and the independent archive record; The archiving analysis module is used to regularly analyze the metadata index and operation log in the SSD array, automatically identify the importance and priority of files, and determine the retention period and archiving strategy of files, so as to achieve data life cycle management, ensure that key data is retained for a long time and obsolete data is processed in a timely manner; The virtual machine environment module is used to periodically create a complete copy of the temporary files currently stored in the SSD array as a benchmark according to the determined file retention period and archiving strategy, compare the copy with the archived data in the long-term storage, and identify the newly added, modified or deleted files in the copy relative to the archived data during the comparison process; and mark the newly added or modified files and transfer them to the long-term storage for archiving; The long-term storage is used for long-term storage of archived files.

Images