Big data hybrid storage system and method, and readable storage medium

Abstract

The present application relates to a big data hybrid storage system and method, and a readable storage medium. The big data hybrid storage system comprises: a first file system, which is used for storing active data; a second file system based on object storage, which is used for storing inactive data; a hybrid storage protocol endpoint, which is used for performing automatic data routing and reading / writing on the first file system and the second file system; and a data synchronization server, which is used for parsing an audit log of the first file system to obtain file usage statistical information, starting a timing task to scan the first file system and the second file system to obtain a corresponding first file list and second file list, and determining, on the basis of the file usage statistical information and the difference between the first file list and the second file list, whether to mirror and synchronize data in the first file system to the second file system. Thus, data storage costs are reduced, and object storage access pressure is alleviated.

Description

Big data hybrid storage system, method and readable storage medium TECHNICAL FIELD

[0001] The present application relates to the technical field of big data storage, in particular to a big data hybrid storage system, method and readable storage medium. BACKGROUND

[0002] In the current big data storage technical solution, data only exists in one storage service, and the advantages of each storage service cannot be fully utilized to form complementarity. For example, the HDFS file system is based on local block storage and has better performance, but the storage cost is higher. The JuiceFS file system is based on object storage and has low storage cost, but cannot run a large number of concurrent jobs, has poor IO performance, and is suitable for batch reading and writing. In actual big data scenarios, the proportion of active data is very small. If data is only stored in the HDFS file system, the storage cost is high. If data is only stored in the object storage, the performance cannot be guaranteed.

[0003] In the related art, the big data storage technology can only store data in one file system, and the advantages of each storage service cannot be fully utilized to form complementarity. Therefore, an effective solution has not been proposed. SUMMARY

[0004] Therefore, it is necessary to provide a big data hybrid storage system, method and readable storage medium to solve the above technical problems.

[0005] In a first aspect, the embodiments of the present application provide a big data hybrid storage system, which comprises:

[0006] A first file system based on local block storage, configured to store active data;

[0007] A second file system based on object storage, configured to store inactive data;

[0008] A hybrid storage protocol end, configured to automatically route reading and writing of data of the first file system and the second file system;

[0009] A data synchronization service end, configured to parse audit logs of the first file system to obtain file usage statistical information, and start a timing task to scan the first file system and the second file system to obtain a first file list and a second file list corresponding to the first file list and the second file list. Based on the file usage statistical information and the difference between the first file list and the second file list, it is determined whether to synchronize the data in the first file system to the second file system.

[0010] In one of the embodiments, the hybrid storage protocol end comprises:

[0011] a data writing module configured to write new data into the first file system;

[0012] a data reading module configured to return a corresponding file list and file content based on a data reading request initiated by a user.

[0013] In one embodiment, the data reading module comprises:

[0014] a file information reading module configured to obtain a first file list of the first file system and a second file list of the second file system after protocol conversion, return a union set of files of directory type in the first file list and the second file list, and return a file list of files of file type in the first file list;

[0015] a file content reading module configured to read a corresponding file in the first file system after protocol conversion, and read a corresponding file in the second file system if the corresponding file does not exist in the first file system.

[0016] In one embodiment, the hybrid storage protocol end further comprises:

[0017] a data deleting module configured to delete files in the first file system and the second file system simultaneously after protocol conversion based on a data deleting request initiated by a user.

[0018] In one embodiment, the data synchronization server comprises:

[0019] a parsing module configured to parse an audit log of the first file system to obtain file usage statistical information in the first file system;

[0020] a scanning and comparing module configured to start a timing task, scan the first file system and the second file system to obtain corresponding file lists, and compare differences between the file lists of the first file system and the second file system;

[0021] a synchronization module configured to determine whether to synchronize data in the first file system to the second file system based on the file usage statistical information in the first file system and the differences between the file lists of the first file system and the second file system.

[0022] In one embodiment, the synchronization module is further configured to:

[0023] if the file usage frequency in the first file system is higher than a preset value, synchronize the files in the first file system to the second file system and delete the corresponding files in the first file system.

[0024] In one of the embodiments, the synchronization module is further configured to:

[0025] If the files in the first file system are not stored in the second file system, the files in the first file system are synchronized to the second file system.

[0026] In one of the embodiments, the second file system is a JuiceFS file system or a JindoData file system.

[0027] In a second aspect, the embodiments of the present application further provide a big data hybrid storage method, applied to the system of the first aspect, and the method comprises:

[0028] Based on the data synchronization service, the audit log of the first file system is parsed to obtain file usage statistics information, and a timing task is started to scan the first file system and the second file system to obtain corresponding first and second file lists.

[0029] Based on the file usage statistics information and the difference between the first and second file lists, it is determined whether to synchronize the data in the first file system to the second file system by using the data synchronization service.

[0030] In a third aspect, the embodiments of the present application further provide a computer readable storage medium, and the storage medium stores a computer program, wherein the computer program is executed by a processor to implement the method of the second aspect.

[0031] The big data hybrid storage system, method and readable storage medium, wherein the big data hybrid storage system comprises a first file system, an object storage-based second file system, a hybrid storage protocol end, a data synchronization service end, the first file system is configured to store active data, the object storage-based second file system is configured to store inactive data, the hybrid storage protocol end is configured to automatically route data read and write of the first file system and the second file system, the data synchronization service end is configured to parse the audit log of the first file system to obtain file usage statistics information, and start a timing task to scan the first file system and the second file system to obtain corresponding first and second file lists, and based on the file usage statistics information and the difference between the first and second file lists, it is determined whether to synchronize the data in the first file system to the second file system, which solves the problem that the big data storage technology can only store data in one file system and cannot fully utilize the advantages of each storage service to form a complementary problem, realizes stable operation of batch jobs in the actual application scenario of big data and realizes data storage cost reduction, and reduces the object storage access pressure.

[0032] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the application will be apparent from the description of illustrative embodiments of the application and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0034] Fig. 1 is a structural block diagram of a big data hybrid storage system in one embodiment;

[0035] Fig. 2 is a flow diagram of file information reading in a big data hybrid storage system in one embodiment;

[0036] Fig. 3 is a flow diagram of file content reading in a big data hybrid storage system in one embodiment;

[0037] Fig. 4 is a flow diagram of file deletion in a big data hybrid storage system in one embodiment;

[0038] Fig. 5 is a flow diagram of file synchronization in a big data hybrid storage system in one embodiment. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the application clearer, the application will be described and illustrated below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and should not be used to limit the application. Based on the embodiments provided by the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the application.

[0040] It is obvious that the drawings described below are only some examples or embodiments of the application, and for those of ordinary skill in the art, the application can be applied to other similar scenarios without creative labor. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the application, some design, manufacture or production changes based on the technical content disclosed in the application are only routine technical means and should not be understood as insufficient disclosure of the application.

[0041] Reference to an "embodiment" in this application means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As used in this application, the term "exemplary" is intended to mean serving as an instance or illustration. The phrase "based on" in some embodiments means, at least in part, based on.

[0042] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the ordinary meanings as understood by one of ordinary skill in the art to which this application pertains. The terms "a", "an", "one", "this", and the like, as used in the present application, do not denote a quantity of one, but rather denote a quantity of at least one. The terms "including", "containing", "having", and the like, as used in the present application, are meant to be open-ended terms that specifically permit the inclusion of one or more steps, components, or elements not listed. The terms "connected", "coupled", "attached", and the like, as used in the present application, are not limited to direct connections, but rather include indirect connections unless expressly specified otherwise. The term "plurality" means two or more. The term "and / or" describes association between or among multiple options, and indicates that three instances can exist, e.g., A and / or B can mean A alone, A and B together, or B alone. The character " / " is generally used to represent "or", unless otherwise specified. The terms "first", "second", "third", and the like, merely identify classes of elements, and do not limit the number of elements in each class.

[0043] The big data hybrid storage system provided by the embodiments of the present application is mainly applied to the field of big data offline data storage, and is suitable for batch data processing scenarios. The big data hybrid storage system involves four key technical entities: a hybrid storage protocol end, a first file system, a second file system, and a data synchronization service end. FIG. 1 is a structural schematic diagram of the big data hybrid storage system according to an embodiment of the present application, which shows the relationship among the four.

[0044] The first file system is a file system based on local block storage, used to store active data; the second file system is a file system based on object storage, used to store inactive data; the hybrid storage protocol end is used for automatic data routing read and write of the first file system and the second file system; the data synchronization service end is used to parse the audit log of the first file system to obtain file usage statistical information; and a timing task is started to scan the first file system and the second file system to obtain a corresponding first file list and a second file list; based on the file usage statistical information and the difference between the first file list and the second file list, it is determined whether to synchronize the data in the first file system to the second file system. Specifically, according to the file usage statistical information, if it is found that the file usage frequency in the first file system is high, for example, higher than a preset value, the data synchronization service end synchronizes the file in the first file system to the second file system and deletes the corresponding file in the first file system, which can relieve the storage pressure of the first file system; and the difference between the first file list and the second file list is compared, if it is found that the file in the first file system is not stored in the second file system, the data synchronization service end synchronizes the file in the first file system to the second file system; the difference between the first file list and the second file list is compared, and the data synchronization service end synchronizes the changed data in the first file system to the second file system.

[0045] The first file system is a HDFS (Hadoop Distributed File System) file system based on local block storage. HDFS is a highly fault-tolerant system suitable for deployment on inexpensive machines. HDFS can provide high-throughput data access and is very suitable for large-scale data sets. It is also a file system protocol.

[0046] The second file system is a file system based on object storage, such as JuiceFS file system, JindoData file system, etc. JuiceFS file system is an open source file system that can be built based on object storage, compatible with HDFS file system, and can seamlessly connect with mainstream computing engines and unlimitedly expand storage space. JindoData file system is a similar file system to JuiceFS, which is a closed-source file system developed by Alibaba Cloud.

[0047] In a big data batch processing scenario, data management is based on folders as a basic unit, and the big data hybrid storage system of the embodiments of the present application is also based on folders for data management, mainly including three major operations: data writing, data synchronization, and data reading. Among them, the HDFS file system and the JuiceFS / JindoData file system are existing technology services for storing actual data, and the hybrid storage system stores active data in the HDFS file system and inactive data in the file system based on object storage, ensuring that active data can be processed efficiently and avoiding excessive object storage pressure causing unstable job operation. Since the HDFS file system only stores part of the data, the data storage cost is reduced. The hybrid storage protocol is compatible with various batch processing technology engines in the big data ecosystem, responsible for writing data into the HDFS file system and reading data from the HDFS file system or the file system based on object storage, and realizing automatic routing of data reading and writing; among them, the data writing operation is relatively simple, and it is directly written into the HDFS file system. The data synchronization server is responsible for mirroring the data of the HDFS file system to the JuiceFS / JindoData file system.

[0048] The big data hybrid storage system designed in the present application, the core idea of which is to store active data in the file system of local block storage and inactive data in the file system of object storage, to balance the storage cost and access performance. Through the file system protocol, data in different file systems can be accessed, and a data synchronization mechanism is designed in the present application. The data synchronization service counts file usage, and decides whether the data is transferred to the object storage according to the data life cycle management strategy, to realize the storage of data in different storage media according to the life cycle, and to ensure the consistency of data. It is the above two key technologies that constitute the hybrid storage system. The hybrid storage system stores data in different hardware cost media, realizes storage cost reduction, and tries to reduce object storage access as much as possible to avoid a large number of batch jobs running in parallel to access object storage, thereby causing the instability of batch jobs. The big data hybrid storage system of the present application can ensure the stable operation of batch jobs in the actual application scenario of big data and realize the reduction of data storage cost, and reduce the access pressure of object storage.

[0049] In one of the embodiments, the hybrid storage protocol end includes a data writing module and a data reading module. The data writing module is used to write new data into the first file system; and the data reading module is used to return the corresponding file list and file content based on a data reading request initiated by a user.

[0050] Specifically, the hybrid storage protocol end mainly implements automatic data routing read and write, including a data writing module and a data reading module. The data writing operation is relatively simple, and the new data is directly written to the HDFS file system. The data reading module is used to query and obtain the corresponding file list and file content in the HDFS file system and JuiceFS / JindoData file system according to the data reading request initiated by the user, and return through the hybrid storage protocol.

[0051] In one embodiment, the data reading module includes a file information reading module, a file content reading module, and a hybrid storage protocol end. The file information reading module is used to obtain the first file list of the first file system and the second file list of the second file system after protocol conversion, and returns the union set of the file types in the first file list and the second file list, and returns the file list of the file type in the first file list. The file content reading module is used to read the corresponding file in the first file system after protocol conversion; if the first file system does not exist, the corresponding file in the second file system is read.

[0052] In a specific embodiment, as shown in FIG. 2, the process of file information reading includes the following contents: the hybrid storage protocol end converts the mfs protocol (hybrid storage protocol) into HDFS protocol or JuiceFS protocol after protocol conversion, and obtains the file list in the HDFS and JuiceFS file systems, and the union set of the file types in the file list, and only the information in the HDFS file system is retained for the file type, and finally the corresponding file list is returned through the hybrid storage protocol.

[0053] In a specific embodiment, as shown in FIG. 3, the process of file content reading includes the following contents: the hybrid storage protocol end converts the mfs protocol (hybrid storage protocol) into HDFS protocol or JuiceFS protocol after protocol conversion, first judges whether the corresponding file content exists in the HDFS file system, if exists, the content information in the HDFS file system is read first, if the corresponding file content does not exist in the HDFS file system, the corresponding file content is read from the JuiceFS file system, and finally the corresponding file content is returned through the hybrid storage protocol.

[0054] In one embodiment, the hybrid storage protocol end further includes a data deletion module, and the data deletion module is used to delete the files in the first file system and the second file system based on the data deletion request initiated by the user after protocol conversion.

[0055] The file deletion process in the embodiment occurs at the hybrid storage protocol end, as shown in FIG. 4. When a user initiates a request to delete data, the hybrid storage protocol end deletes the file in the HDFS file system and the JuiceFS file system simultaneously after protocol conversion, thereby avoiding errors in reading the file content due to inconsistency of the files.

[0056] In one of the embodiments, the data synchronization server comprises an analysis module, a scanning and comparison module, and a synchronization module. The analysis module is configured to analyze the audit log of the first file system to obtain file usage statistics in the first file system. The scanning and comparison module is configured to start a timing task, scan the first file system and the second file system to obtain corresponding file lists, and compare the differences between the file lists of the first file system and the second file system. The synchronization module is configured to determine whether to synchronize the data in the first file system to the second file system based on the file usage statistics in the first file system and the differences between the file lists of the first file system and the second file system.

[0057] In the embodiment, the file synchronization process occurs at the data synchronization server and is completed by the Sync service. As shown in FIG. 5, the Sync service analyzes the audit log in the HDFS file system to form file usage statistics, starts a timing task to scan the HDFS file system and the JuiceFS file system, obtains corresponding file lists, compares the differences, and determines whether to synchronize the data to the JuiceFS file system and whether to delete the file in the HDFS file system based on the file usage statistics and the differences.

[0058] In one of the embodiments, the synchronization module is further configured to synchronize the file in the first file system to the second file system and delete the corresponding file in the first file system if it is found that the file usage frequency in the first file system is high, for example, higher than a preset value, based on the file usage statistics. In this way, the storage pressure of the first file system can be relieved.

[0059] In one of the embodiments, the synchronization module is further configured to compare the differences between the first file list and the second file list, synchronize the file in the first file system to the second file system if it is found that the file in the first file system is not stored in the second file system, and synchronize the changed data in the first file system to the second file system by comparing the differences between the first file list and the second file list.

[0060] Further, the data synchronization service also determines whether to synchronize the data into the JuiceFS file system and whether to delete the file in the HDFS file system according to the data life cycle management configuration.

[0061] The central idea of the big data hybrid storage system designed in the application is similar to the cache technology, such as the cache technology of JuiceFS / JindoData, but has the following advantages:

[0062] The cache technology is managed according to the cache eviction algorithm, and cannot manage the content that needs to be cached on demand, so it cannot match the life cycle of data.

[0063] JuiceFS / JindoData also has an independent cache cluster, but the independent cache cluster is separated from the computing task, and the computing is non-localized, and the performance is not as good as that of the HDFS cluster when processing active data.

[0064] The data cache of JuiceFS / JindoData is passive cache, and if there is no corresponding data locally when the user uses the data for the first time, the data in the object storage will be downloaded to the local, and the local cache can be used for the second time. That is, the cache hit rate may not be high, and the performance is not as good as that of the HDFS cluster when processing active data.

[0065] Based on the same inventive concept, the embodiments of the application also provide a big data hybrid storage method applied to the big data hybrid storage system as described in any of the above embodiments, and the method comprises the following contents: based on the data synchronization service, parsing the audit log of the first file system to obtain file usage statistical information; and starting a timing task to scan the first file system and the second file system to obtain corresponding first file list and second file list; based on the file usage statistical information and the difference between the first file list and the second file list, determining whether to synchronize the data in the first file system to the second file system by using the data synchronization service.

[0066] The big data hybrid storage method in the embodiment stores active data in a first file system and stores inactive data in a second file system based on object storage. The first file system is an HDFS file system, and the second file system is a JuiceFS file system or a JindoData file system. A hybrid storage protocol is used to automatically route data read and write of the first file system and the second file system. An audit log of the first file system is parsed by using a data synchronization service to obtain file usage statistical information. A timing task is started to scan the first file system and the second file system to obtain a first file list and a second file list. Based on the file usage statistical information and a difference between the first file list and the second file list, it is determined whether to synchronize data in the first file system to the second file system.

[0067] Further, the hybrid storage protocol is also used for data writing and data reading, including writing new data into the first file system. Based on a data reading request initiated by a user, a corresponding file list and file content are returned.

[0068] Further, the hybrid storage protocol is also used to obtain, after protocol conversion, a first file list of the first file system and a second file list of the second file system, and return a union set of files of directory type in the first file list and the second file list, and return a file list of file type in the first file list.

[0069] Further, the hybrid storage protocol is also used to read a corresponding file in the first file system after protocol conversion. If the corresponding file does not exist in the first file system, a corresponding file in the second file system is read.

[0070] Further, the hybrid storage protocol is also used to delete files in the first file system and the second file system after protocol conversion based on a data deletion request initiated by a user.

[0071] Further, the data synchronization service is also used to parse an audit log of the first file system to obtain file usage statistical information in the first file system. A timing task is started to scan the first file system and the second file system to obtain a corresponding file list and compare a difference between file lists of the first file system and the second file system. Based on the file usage statistical information in the first file system and the difference between the file lists of the first file system and the second file system, it is determined whether to synchronize data in the first file system to the second file system.

[0072] Further, the data synchronization service is further configured to synchronize the file in the first file system to the second file system and delete the corresponding file in the first file system when the file in the first file system has a frequency of use higher than a preset value.

[0073] Further, the data synchronization service is further configured to synchronize the file in the first file system to the second file system when the file in the first file system is not stored in the second file system.

[0074] The big data hybrid storage method of the embodiment of the application manages data based on folders, mainly including three operations: data writing, data synchronization, and data reading. Among them, the HDFS file system and the JuiceFS / JindoData file system are existing technical services for storing actual data. The hybrid storage system stores active data in the HDFS file system and inactive data in the file system based on object storage, ensures efficient processing of active data, avoids excessive object storage pressure causing unstable operation of jobs, and reduces data storage costs due to the fact that the HDFS file system only stores part of the data. The hybrid storage protocol is compatible with various batch processing technology engines in the big data ecosystem, is responsible for writing data into the HDFS file system and reading data from the HDFS file system or the file system based on object storage, and realizes automatic routing of data reading and writing; among them, the data writing operation is relatively simple, and the data is directly written into the HDFS file system. The data synchronization server is responsible for mirroring the data of the HDFS file system to the JuiceFS / JindoData file system.

[0075] The big data hybrid storage method designed in the application has the central idea of storing active data in local block storage and inactive data in object storage to balance storage costs and access performance. Through the file system protocol, data in different file systems can be accessed. The application designs a data synchronization mechanism. The data synchronization service counts file usage and decides whether to transfer data to object storage according to the data life cycle management strategy, realizes data storage in different storage media according to the life cycle, and ensures data consistency. It is the above two key technologies that constitute the hybrid storage system. The hybrid storage system stores data in different hardware cost media, realizes storage cost reduction, and minimizes object storage access to avoid a large number of batch jobs running in parallel to access object storage, thereby causing instability of batch jobs. The big data hybrid storage system of the application can ensure stable operation of batch jobs in actual application scenarios of big data and realize reduction of data storage costs, and reduce object storage access pressure.

[0076] In one embodiment, a computer readable storage medium is provided, having stored thereon a computer program, which when executed by a processor implements the steps of any of the above described embodiments of the big data hybrid storage system.

[0077] A person of ordinary skill in the art can understand that all or part of the processes in the above described embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the computer program can include the processes of the above described embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, the RAM can be in a variety of forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0078] Any combination of the technical features of the above described embodiments can be made, and in order to make the description simple, all possible combinations of the technical features in the above described embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0079] The above described embodiments only express several implementation manners of the present application, the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for a person of ordinary skill in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A big data hybrid storage system, characterized in that, The system comprises: a first file system based on local block storage, used for storing active data; a second file system based on object storage, used for storing inactive data; a hybrid storage protocol end, used for automatically routing data read and write of the first file system and the second file system; a data synchronization service end, used for parsing an audit log of the first file system to obtain file usage statistical information, and starting a timing task to scan the first file system and the second file system to obtain a first file list and a second file list corresponding thereto, and determining whether to synchronize data in the first file system to the second file system based on the file usage statistical information and a difference between the first file list and the second file list.

2. The system of claim 1, wherein, The hybrid storage protocol end comprises: a data writing module, used for writing new data into the first file system; a data reading module, used for returning a corresponding file list and file content based on a data reading request initiated by a user.

3. The system of claim 2, wherein, The data reading module comprises: a file information reading module, used for obtaining a first file list of the first file system and a second file list of the second file system after protocol conversion, returning a union set of files of directory type in the first file list and the second file list, and returning a file list of files of file type in the first file list; a file content reading module, used for reading a corresponding file in the first file system after protocol conversion, and reading a corresponding file in the second file system if the corresponding file does not exist in the first file system.

4. The system of claim 1, wherein, The hybrid storage protocol end further comprises: a data deleting module, used for deleting files in the first file system and the second file system simultaneously after protocol conversion based on a data deleting request initiated by a user.

5. The system of claim 1, wherein, The data synchronization service end comprises: an analysis module, used for analyzing an audit log of the first file system to obtain file usage statistical information in the first file system; a scanning and comparing module, used for starting a timing task to scan the first file system and the second file system to obtain corresponding file lists, and comparing a difference between the file lists of the first file system and the second file system; a synchronization module, used for determining whether to synchronize data in the first file system to the second file system based on the file usage statistical information in the first file system and the difference between the file lists of the first file system and the second file system.

6. The system of claim 5, wherein, The synchronization module is further used for: if a file usage frequency in the first file system is higher than a preset value, synchronizing the file in the first file system to the second file system, and deleting a corresponding file in the first file system.

7. The system of claim 5, wherein, The synchronization module is further used for: if a file in the first file system is not stored in the second file system, synchronizing the file in the first file system to the second file system.

8. The system of claim 1, wherein, The second file system is a JuiceFS file system or a JindoData file system.

9. A big data hybrid storage method applied to the system of any one of claims 1-8, characterized in that, The method comprises: Based on the data synchronization service, parse the audit log of the first file system to obtain file usage statistics information; and start a timing task to scan the first file system and the second file system to obtain a corresponding first file list and a second file list; Based on the file usage statistics information and the difference between the first file list and the second file list, determine whether to synchronize the data in the first file system to the second file system by using the data synchronization service.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the method of claim 9.

Images