A method and device for quickly accessing data files

Abstract

The present invention discloses a method for rapid access to data files, comprising the following steps: S1, receiving a data file and preprocessing the data file; S2, performing quick freezing and storage processing on the data file according to storage requirements and compression ratio; S3, performing quick thawing and reading processing on the data file. The present invention also provides a rapid access device used in the above-mentioned method for rapid access to data files. The present invention adopts technologies such as hot and cold data tiered storage, quick freezing and thawing of data, data compression and query optimization, as well as a Blu-ray storage medium that takes both cost and capacity into consideration. When accessing data files, the present invention does not require a disk directory to search level by level, but quickly locates the file storage location in the database through keywords and reads the content of the data file. This not only provides a storage method with lower cost, higher security and suitable for long-term data storage, but also enables a large number of data files to be stored long-term or permanently.

Description

Technical Field

[0001] The present invention relates to the field of processing, storing and reading a large number of data files, and more particularly to a method and device for quickly accessing data files. Background Art

[0002] For industries such as government, insurance, healthcare, education, and finance, a large number of files require long-term or permanent storage to safeguard the legitimate rights and interests of users and businesses, protect user privacy, and provide sufficient evidence for complaints, legal proceedings, and historical research. For example, in the insurance industry, documents such as insurance application images and signature photos must be stored for the entire policy period; lifetime insurance business files must be stored for at least 50 years. In the healthcare industry, national regulations require that data files be stored for at least 15-30 years. Database backups at financial institutions require permanent storage, and bank window service data must be stored until the end of the business.

[0003] As time passes and business becomes more digital, these long-term or permanently stored data files continue to be generated, and the rate of generation continues to accelerate as the information and digitalization levels of various industries advance. For example, banks need to photograph and save every ID document every time they conduct business, generating a large number of files every day. Insurance companies and insurance intermediaries also generate data files such as the personal information of policyholders and insured persons, as well as audio and video recordings, photos, and other data files for each transaction.

[0004] For business scenarios where large numbers of files need to be quickly archived, stored, and read, it is necessary not only to solve the problem of storing massive data files, but also to solve how to quickly store these files and how to quickly access these files when in use. This has become a common demand of relevant government departments, enterprises, and industry users.

[0005] To meet this demand, there are currently three main types of solutions, namely disk, tape and optical disk storage.

[0006] Disk storage solutions store these data files on disk devices, such as dedicated storage appliances and servers. This solution provides relatively fast query performance when accessing data files. However, due to device limitations and the influence of the storage medium, this solution faces scalability bottlenecks, failing to provide sufficient storage space and scalability for the growing number of files. Furthermore, since the devices typically use disks for storage, while having fast read and write speeds, their lifespan is limited to three to five years, making them unsuitable for storing archived data files for decades. Furthermore, disk procurement costs are relatively high, significantly increasing the capital investment of government departments and businesses.

[0007] Tape storage solutions use tape as the storage medium for storing data files. A tape drive and its controller comprise a storage device. When recording or reading data, tape drives the tape relative to a magnetic head, using the magnetic head to perform electromagnetic conversion to achieve sequential recording or reading. Compared to magnetic disks and optical disks, tape storage has a moderate access speed. However, tape storage media is susceptible to damage from electromagnetic radiation, and the data signal on the tape weakens over time, requiring regular inspection and refresh. Over time, tape wears out, becomes damp, and moldy, resulting in a limited lifespan. There are numerous tape specifications and they are incompatible with each other. If older tape drives are retired, the data on the tapes will become unreadable. Furthermore, as existing storage technology matures, the price advantages of tape storage are gradually disappearing.

[0008] Optical disc storage uses optical discs as the storage medium for data files. Given their high recording density and contactless read / write capabilities, optical discs offer a large storage capacity, meeting the requirements for storing large amounts of data files. They are also highly reliable and environmentally resistant, resisting damage and data loss. Data can be stored for a long period of time, with a lifespan of 10 to 50 years. However, when reading data, optical discs rotate at a lower speed than magnetic disks, and data correction and optical disc wear can affect speed. Therefore, access speeds for optical disc storage solutions are slower than those for magnetic disks and tapes.

[0009] Looking at the three options above, disk storage offers relatively fast read and write speeds but a short lifespan. Tape storage offers moderate access speeds but a short lifespan, poor reliability, and strict requirements for its use and storage environment. Optical discs offer a long lifespan and high reliability, but suffer from the slowest data access speeds. Therefore, none of these options can meet the current demand for long-term storage and fast access to large amounts of data files.

[0010] In this case, in order to take into account the storage space, access performance and overall cost of a large number of data files, it is necessary to design and provide a fast access device for data files. By adopting hot and cold tiered storage, which uses disks to store hot data and optical disks to store cold data, data quick freezing and thawing, data compression and query optimization, a large number of data files can be quickly stored and read at the optimal cost. At the same time, the system or device also needs to have strong compatibility, simple design, flexible deployment, convenient operation, low procurement and use costs, and strong scalability. It can be easily promoted and used on a large scale to meet the needs of long-term storage, fast access and cost reduction of large amounts of data files, and improve the availability, reliability and scalability of the data file fast storage and reading device for large amounts of data files. Summary of the Invention

[0011] The object of the present invention is to provide a method for quickly accessing data files to solve the problems raised in the above background technology. The present invention provides the following technical solutions:

[0012] A method for quickly accessing a data file comprises the following steps:

[0013] S1. Receive a data file and pre-process the data file, which includes the following sub-steps:

[0014] S11, setting an access frequency value for the received data file;

[0015] S12, extracting file information of the data file;

[0016] S13. Calculate the data file compression ratio based on the file information and determine the storage method and storage path;

[0017] S2. Perform quick freezing storage on the data files based on storage requirements and compression ratio, which includes the following sub-steps:

[0018] S21, generating an index file of the data file;

[0019] S22. compressing the data file according to the data file compression ratio;

[0020] S23. Storing the compressed data file according to the storage method and path;

[0021] S3, data file fast thawing and reading processing, including the following sub-steps:

[0022] S31. Obtaining data file information from a user query request, and obtaining a data file index matching the data file based on the data file information;

[0023] S32. Query and extract the compressed data file from the storage module according to the data file index;

[0024] S33. Decompressing the extracted compressed data file according to the data file compression ratio in the data file index;

[0025] The format of the data file index file includes the following fields:

[0026] Field 1: data file compression ratio C;

[0027] Field 2: storage path;

[0028] Field 3: access frequency value;

[0029] Field 4: Data file information.

[0030] The present invention improves the performance and efficiency of storing and reading a large number of data files by extracting data file information, quickly freezing and thawing and reading data files, compressing and decompressing files, differentially storing hot and cold data, and optimizing indexes. The present invention adopts technologies such as tiered storage of hot and cold data, quick freezing and thawing of data, data compression and query optimization, as well as Blu-ray storage media that takes both cost and capacity into consideration. When accessing data files, there is no need for disk directory level-by-level retrieval, and the file storage location is quickly located in the database through keywords, and the content of the data file is read. This not only provides storage with lower cost, higher security and suitability for long-term data storage, allowing a large number of data files to be stored long-term or permanently, but also solves the problems of insufficient storage space, high usage cost, and slow reading speed in current data file storage.

[0031] Preferably, the data file compression rate is calculated and obtained by using a Rapid Dynamic Compression (RDC) model, wherein the RDC model is:

[0032] C = log max(N，n) min(N,n)*100%;

[0033] Where C is the data file compression ratio; N is the storage limit of the storage module used to store the data file; n is the storage condition value of the data file;

[0034] The storage method is determined according to the data storage requirements in the file information, and the storage path is determined according to the storage module corresponding to the data storage requirements;

[0035] The storage module can dynamically sort the stored data files in sequence according to the access frequency values ​​of the data files; the sequence sorting rule is to sort according to the access frequency.

[0036] Preferably, the step S3.2 of searching and extracting the compressed data file from the storage module according to the data file index specifically includes the following steps:

[0037] S321. Obtaining a data file compression ratio from the data file information obtained in the user query request;

[0038] S322, determining an index file for the data file based on the data file compression ratio and field 1 in the index file;

[0039] S323. Determine the storage path of the data file according to field 2 in the index file;

[0040] S324. Find the storage module storing the data file according to field 2 in the index file;

[0041] S325. Obtain the data file according to field 4 in the index file.

[0042] Preferably, obtaining the data file according to field 4 of the index file in step S325 specifically includes the following steps:

[0043] S3251. Extract statistical information on the access frequency of the data file from the storage module, where the statistical information is the maximum and minimum values ​​of the access frequency of the data file, and calculate the median value of the access frequency based on the maximum and minimum values.

[0044] S3252: Divide the storage module into a storage area 1 having an access frequency greater than the median value of the access frequency and a storage area 2 having an access frequency less than the median value of the access frequency according to the median value of the access frequency;

[0045] S3253: Compare field 3 in the index file with the median value of the access frequency to determine the storage area;

[0046] S3254. Obtain data file.

[0047] The present invention also provides a fast access device used in the above-mentioned method for fast accessing data files, comprising a data receiving module, a hot storage pool, a data processing module, a cold storage pool, a thawing module, a fast query information storage unit, and an output unit; the data receiving module is connected to a data file source via a data line, and the data receiving module is connected to the hot storage pool and the data processing module via a data bus; the hot storage pool is connected to the data receiving module and the data processing module via a data bus; the data processing module is connected to the data receiving module, the hot storage pool, the cold storage pool, and the fast query information storage unit via a data bus; the cold storage pool is connected to the data processing module and the thawing module via a data bus; the thawing module is connected to the cold storage pool, the hot storage pool, and the fast query information storage unit via a data bus; the fast query information storage unit is connected to the data processing module, the thawing module, and the output unit via a data bus;

[0048] The output unit is used to separate and obtain query conditions from the user request received by the file fast storage and reading device, and send the query conditions to the fast query information storage unit; then read the query results corresponding to the query conditions from the hot storage pool and output them, thereby completing the fast reading of the data file;

[0049] When the result file exists in the hot storage pool, the query result is "true", and the output unit reads the result data file from the hot storage pool and outputs it; when the result file does not exist in the hot storage pool, the query result is "false", and the output unit outputs "query result does not exist".

[0050] Furthermore, the data receiving module includes a stream data access unit and a batch data access unit;

[0051] The stream data access unit is connected to the data processing module via a data bus, and is used to provide a connection interface for the stream data file or receive the stream data file like a stream, and at the same time identify the size, date, type, MD5 value, storage directory and other information of the stream data file, and transmit the received data file to the data processing module; wherein, the stream data access unit includes a plurality of stream connectors, and the connectors are connected in parallel via the data bus; the stream connectors can be created on demand by the stream data access unit when the stream data file is accessed, or can be created in advance and allocated on demand by the stream data access unit when the stream data file is accessed;

[0052] The batch data access unit is connected to the hot storage pool via a data bus, and is used to provide a connection interface for batch data files, and access batch data files in batches, while identifying information such as the size, date, type, MD5 value, storage directory, etc. of the batch data files, and transferring the accessed data files to the hot storage pool;

[0053] The batch data access unit includes multiple batch connectors, which are connected in parallel via a data bus. The batch connectors can be created on demand by the stream data access unit when batch data files are accessed, or they can be created in advance and grouped according to the type of data files, and allocated on demand by the stream data access unit when batch data files are accessed.

[0054] Further, the hot storage pool includes a batch file storage unit and a result file storage unit;

[0055] The batch file storage unit is connected to the batch data access unit and the data processing module via a data bus, and is used to temporarily store batch data files while waiting for processing by the data processing module, thereby alleviating the data processing pressure of the data processing module;

[0056] The result file storage unit is connected to the thawing module and the output unit via a data bus, and is used to receive and temporarily store the result data file, thereby improving the data file reading speed when the user accesses it and achieving fast reading;

[0057] Wherein, the batch file storage unit and the result file storage unit both use magnetic media as storage media, thereby providing efficient data storage and data file access performance while taking into account costs, and improving the efficiency of data file transmission, storage and reading.

[0058] Furthermore, the data processing module includes a metadata processing unit, a quick freezing processing unit and a compression unit;

[0059] The metadata processing unit is used to extract metadata from the data file and transmit it to the fast query information storage unit;

[0060] The quick freezing processing unit is used to quick freeze the data files, that is, to classify the data files by adding data file tags and adding access rights;

[0061] The compression unit is used to compress and package the extracted metadata and the frozen data files at the file directory level, achieving an operation similar to "quick freezing" the files in the directory, and then transfer the compressed and packaged data files to the cold storage pool for storage, thereby reducing the file size through multi-file directory compression, thereby reducing the amount of writing to the storage medium, and ultimately achieving fast file writing;

[0062] The metadata processing unit, quick freezing processing unit and compression unit are all connected to the data receiving module, hot storage pool, cold storage pool and quick query information storage unit through a data bus; the data files include stream data files and batch data files. The stream data files are directly processed after being connected by the stream connector, and the batch data files are first cached in the batch storage unit and batch processed according to a preset data volume threshold.

[0063] Furthermore, the cold storage pool includes a data storage unit, a backup unit, a spare storage unit, and a management unit;

[0064] The data storage unit is used to store the data files processed by the data processing module for a long term or permanently;

[0065] The backup unit is used to back up the data files stored in the data storage unit, so that when the data storage unit is abnormal or fails, the data files can be quickly restored, thereby improving the reliability;

[0066] The backup unit is used to serve as a backup storage resource for the data storage unit, so that when the data storage unit fails or is damaged, the storage task and the stored data files can be quickly switched or migrated to the backup storage unit, thereby ensuring the continuity of the operation of the device and the integrity of the data storage;

[0067] The management unit is used to manage stored data files, and schedule and manage the collaborative work, storage task switching, data migration, etc. among the data storage unit, the backup unit and the standby storage unit;

[0068] Among them, the data storage unit, the backup unit and the standby unit are all connected to the data processing module and the thawing module through a data bus, and all use Blu-ray discs as storage media. Blu-ray storage media can store data files for more than 50 years, thereby realizing long-term or permanent storage of a large number of data files and reducing storage costs.

[0069] Furthermore, the thawing module includes a decompression unit and a thawing processing unit;

[0070] The decompression unit is used to obtain and obtain a corresponding data file from the cold storage pool according to the request metadata corresponding to the user query condition stored in the fast query information storage unit, and decompress the obtained data file to obtain a decompressed data file;

[0071] The thawing processing unit is used to thaw the decompressed data file to obtain a result data file after thawing; then, when the result data file exists in the cold storage pool, the result data file is uploaded to the hot storage pool for user access and download; when the result data file does not exist in the cold storage pool, a null value is returned;

[0072] The decompression unit and the thawing processing unit are both connected to the cold storage pool, the hot storage pool and the fast query information storage unit via a data bus.

[0073] Compared with the prior art, the advantages of the present invention are:

[0074] 1. The present invention improves the performance and efficiency of storing and reading large amounts of data files by extracting data file information, rapidly freezing and thawing data, compressing and decompressing files, differentially storing hot and cold data, and optimizing indexes. The present invention utilizes technologies such as tiered storage of hot and cold data, rapid freezing and thawing of data, data compression, and query optimization, as well as cost-effective and capacity-optimized Blu-ray storage media. Furthermore, when accessing data files, the present invention eliminates the need for hierarchical disk directory searches and rapidly locates file storage locations in the database using keywords, allowing for the reading of data file contents. This not only provides a lower-cost, more secure, and long-term data storage solution, enabling the long-term or permanent storage of large amounts of data files, but also addresses current data file storage challenges such as insufficient storage space, high costs, and slow read speeds.

[0075] 2. Flexible data file processing: According to business needs, data files can be received and processed in real time like a pipeline, or they can be processed in batches at regular intervals and in fixed quantities, effectively improving the efficiency and flexibility of data file processing and making it more applicable.

[0076] 3. Massive storage space that balances reliability and cost: This system provides devices suitable for long-term or permanent storage of large amounts of data files, with storage space that can be expanded on demand based on business growth. This reduces storage costs while ensuring the reliability and security of long-term or permanent data storage. This system upgrades the storage technology and capabilities for large amounts of data files.

[0077] 4. Improve storage space utilization and transmission speed: By compressing data files, the storage space usage of data files and the transmission speed of data files within the system are reduced.

[0078] 5. Fast data file reading capability: Quickly locate the storage location of data files through metadata, and use hot storage to provide cache support for large-scale data file downloads, greatly improving the speed of querying and reading large amounts of data files.

[0079] 6. The fast access device for data files of the present invention selects different media as storage media according to the processing requirements and processes of different data files during design, and has simple design, flexible deployment, and convenient operation. While improving the storage and reading efficiency of a large number of files, it can also help customers lower the usage threshold, quickly complete deployment, and reduce the pressure of daily operation and maintenance, and obtain the best database service experience such as data file storage, processing and access at a low cost, meeting the short-term and long-term needs of business growth. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 A schematic structural diagram of a device for quickly accessing data files according to an embodiment of the present invention;

[0081] Figure 2 Schematic diagram of the structure of a data receiving module according to an embodiment of the present invention;

[0082] Figure 3 This is a schematic structural diagram of a heat storage pool according to an embodiment of the present invention;

[0083] Figure 4 Schematic diagram of the structure of a data processing module according to an embodiment of the present invention;

[0084] Figure 5 This is a schematic structural diagram of a cold storage pool according to an embodiment of the present invention;

[0085] Figure 6 This is a structural diagram of a thawing module according to an embodiment of the present invention;

[0086] Figure 7 The figure is a flowchart of a fast access to a data file according to an embodiment of the present invention. DETAILED DESCRIPTION

[0087] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0088] In the description of the present invention, “plurality” means two or more, unless otherwise clearly defined.

[0089] Example 1

[0090] like Figure 1 As shown, this embodiment provides a fast access device for data files, including a data receiving module 100, a hot storage pool 200, a data processing module 300, a cold storage pool 400, a thawing module 500, a fast query information storage unit 600 and an output unit 700.

[0091] The data receiving module 100 is connected to the data file source via a data line, and is connected to the hot storage pool 200 and the data processing module 300 via a data bus; the hot storage pool 200 is connected to the data receiving module 100 and the data processing module 300 via a data bus; the data processing module 300 is connected to the data receiving module 100, the hot storage pool 200, the cold storage pool 400, and the quick query information storage unit 600 via a data bus; the cold storage pool 400 is connected to the data processing module 300 and the thawing module 500 via a data bus; the thawing module 500 is connected to the cold storage pool 400, the hot storage pool 200, and the quick query information storage unit 600 via a data bus; the quick query information storage unit 600 is connected to the data processing module 300, the thawing module 500, and the output unit 700 via a data bus;

[0092] The output unit 700 is used to separate and obtain query conditions from the user request received by the file fast storage and reading device, and send the query conditions to the fast query information storage unit 600; then read the query results corresponding to the query conditions from the hot storage pool 200 and output them, thereby completing the fast reading of the data file;

[0093] When the result file exists in the thermal storage pool 200, the query result is "true", and the output unit 700 reads the result data file from the thermal storage pool 200 and outputs it; when the result file does not exist in the thermal storage pool 200, that is, the read return result value is a null value, the query result is "false", and the output unit 700 outputs "query result does not exist".

[0094] like Figure 2 As shown, the data receiving module 100 includes a stream data access unit 101 and a batch data access unit 102;

[0095] The stream data access unit 101 is connected to the data processing module 300 via a data bus, and is used to provide a connection interface for stream data files or receive stream data files like flowing water, identify information such as the size, date, type, MD5 value, storage directory, etc. of the stream data files, and transmit the received data files to the data processing module; wherein the stream data access unit 101 includes a plurality of stream connectors, which are connected in parallel via a data bus; the stream connectors can be created on demand by the stream data access unit 101 when a stream data file is accessed, or can be created in advance and allocated on demand by the stream data access unit 101 when a stream data file is accessed;

[0096] The batch data access unit 102 is connected to the hot storage pool 200 via a data bus, and is used to provide a connection interface for batch data files, access batch data files in batches, identify the size, date, type, MD5 value, storage directory and other information of the batch data files, and transfer the accessed data files to the hot storage pool 200;

[0097] The batch data access unit 102 includes multiple batch connectors, which are connected in parallel via a data bus. The batch connectors can be created on demand by the stream data access unit 101 when batch data files are accessed, or they can be created in advance and grouped according to the type of data files, and allocated on demand by the stream data access unit 101 when batch data files are accessed.

[0098] like Figure 3 As shown, the hot storage pool 200 includes a batch file storage unit 201 and a result file storage unit 202;

[0099] The batch file storage unit 201 is connected to the batch data access unit 102 and the data processing module 300 via a data bus, and is used to temporarily store batch data files while waiting for processing by the data processing module 300, thereby alleviating the data processing pressure of the data processing module 300;

[0100] The result file storage unit 202 is connected to the thawing module 500 and the output unit 700 via a data bus, and is used to receive and temporarily store the result data file, thereby improving the data file reading speed when the user accesses it and achieving fast reading;

[0101] The batch file storage unit 201 and the result file storage unit 202 both use magnetic media as storage media, thereby providing efficient data storage and data file access performance while taking into account costs, thereby improving the efficiency of data file transmission, storage and reading.

[0102] like Figure 4 As shown, the data processing module 300 includes a metadata processing unit 301, a quick freezing processing unit 302 and a compression unit 303;

[0103] The metadata processing unit 301 is used to extract metadata from the data file and transmit it to the quick query information storage unit 600;

[0104] The quick freezing processing unit 302 is used to quick freeze the data files, that is, to classify the data files by adding data file tags and adding access rights;

[0105] The compression unit 303 is used to compress and package the extracted metadata and frozen data files based on the file directory level, achieving an operation similar to "freezing" the files in the directory, and then transfer the compressed and packaged data files to the cold storage pool 400 for storage. This reduces the file size through multi-file directory compression, thereby reducing the amount of writing to the storage medium and ultimately achieving fast file writing.

[0106] The metadata processing unit 301, the quick freezing processing unit 302 and the compression unit 303 are all connected to the data receiving module 100, the hot storage pool 200, the cold storage pool 400 and the quick query information storage unit 600 through a data bus; the data files include stream data files and batch data files. The stream data files are directly processed after being connected by the stream connector 101, and the batch data files are first cached in the batch storage unit 201 and batch processed according to a preset data volume threshold.

[0107] like Figure 5 As shown, the cold storage pool 400 includes a data storage unit 401, a backup unit 402, a backup unit 403, and a management unit 404;

[0108] The data storage unit 401 is used to store the data files processed by the data processing module 300 for a long term or permanently;

[0109] The backup unit 402 is used to back up the data files stored in the data storage unit 401, so that when the data storage unit 401 is abnormal or fails, the data files can be quickly restored, thereby improving the reliability;

[0110] The backup unit 403 is used to provide a backup storage resource for the data storage unit 401, so that when the data storage unit 401 fails or is damaged, storage tasks and stored data files can be quickly switched or migrated to the backup storage unit, thereby ensuring the continuity of the device operation and the integrity of data storage;

[0111] The management unit 404 is used to manage stored data files and schedule and manage the collaborative work, storage task switching, data migration, etc. among the data storage unit 401, the backup unit 402, and the backup unit 403;

[0112] Among them, the data storage unit 401, the backup unit 402 and the backup unit 403 are all connected to the data processing module 300 and the thawing module 500 through a data bus, and all use Blu-ray discs as storage media. Blu-ray storage media can store data files for more than 50 years, thereby realizing long-term or permanent storage of a large number of data files and reducing storage costs.

[0113] like Figure 6 As shown, the thawing module 500 includes a decompression unit 501 and a thawing processing unit 502;

[0114] The decompression unit 501 is configured to obtain the request metadata corresponding to the user query condition stored in the quick query information storage unit 600, obtain the corresponding data file from the cold storage pool 400, and decompress the obtained data file to obtain a decompressed data file;

[0115] The thawing processing unit 502 is used to thaw the decompressed data file to obtain a result data file after thawing; then, if the result data file exists in the cold storage pool 400, the result data file is uploaded to the hot storage pool 200 for user access and download; if the result data file does not exist in the cold storage pool 400, a null value is returned;

[0116] The decompression unit 501 and the thawing processing unit 502 are both connected to the cold storage pool 400 , the hot storage pool 200 and the fast query information storage unit 600 via a data bus.

[0117] The index file format is the format of the index file of the data file, wherein:

[0118] Field 1: Data file compression ratio C, which is the compression ratio value of the compressed data file. The data file compression ratio C is calculated based on the file information of the received data file using a rapid dynamic compression (RDC) model and is used to determine the storage module of the data file.

[0119] Wherein, the RDC model is:

[0120] C = log max(N，n) min(N,n)*100%;

[0121] Where C is the data file compression ratio; N is the storage limit of the storage module used to store the data file; n is the storage condition value of the data file;

[0122] Field 2: Storage path, which is the path corresponding to the storage module where the data file is stored, and is determined according to the storage module corresponding to the data storage requirement.

[0123] Field 3: Access frequency value, which is a statistical value of the access frequency of the data file stored in the storage module. The access frequency value is obtained from the statistical information of the access frequency of the data file stored in the storage module.

[0124] Field 4: Data file information, which is the description information of the data file and is used to provide a specific description of the data file, such as the data file name, size, date, type, MD5 value, metadata, business attributes, storage conditions, etc.

[0125] The data file rapid storage process includes the following steps:

[0126] S1. Receive data files and pre-process the data files: Receiving data files includes: real-time reception of streaming data and real-time reception of batch data. For real-time streaming data, the system provides a streaming data interface to connect the streaming data to the system, and selects different processing methods for cumulative processing based on the business attributes of the streaming data. When the business attribute of the streaming data is time, the streaming data is accumulated according to the time period, and the data file of each time period is converted into a unified data file format of the system; when the business attribute of the streaming data is an event, the streaming data file is classified and processed according to the event and the data file format is uniformly converted, and the start and end of the event are marked. For batch historical data, through the data file access interface, such as JDBC, ODBC, etc., when receiving the data file, it is determined whether the format of the data file matches the system storage format. If it does not match, the format is converted and then batched; if it matches, it is directly batched.

[0127] S11. Set an access frequency value for the received data file: the initial value of the access frequency value of the data file is "0", which is set by means of a data file note; as the access frequency of the data file increases, the initial value of the access frequency value will also increase, and access frequency statistics of the data file will be generated.

[0128] S12. Extract file information of the data file: The file information of the extracted data file includes: the name, size, date, type, MD5 value, metadata, business attributes, storage conditions, etc. of the data file.

[0129] Calculating a data file compression ratio based on the file information and determining a storage method and a storage path: the data file compression ratio is calculated and obtained using a Rapid Dynamic Compression (RDC) model; the storage method is determined based on data storage requirements in the file information, and the storage path is determined based on a storage module corresponding to the data storage requirements;

[0130] The storage module stores the statistical information of the access frequency of the data files, and can dynamically sort the stored data files in sequence according to the access frequency values ​​of the data files, and the sequence sorting rule is to sort according to the access frequency.

[0131] S2. Perform quick freezing storage on the data files based on storage requirements and compression ratio:

[0132] S21. Generate an index file for the data file: The format of the index file for the data file includes the following fields:

[0133] Field 1: data file compression ratio C;

[0134] Field 2: storage path;

[0135] Field 3: access frequency value;

[0136] Field 4: Data file information.

[0137] S22. compressing the data file according to the data file compression ratio;

[0138] S23. Storing the compressed data file according to the storage method and path;

[0139] S24. The data file is stored successfully.

[0140] The data file reading process includes the following steps:

[0141] S3, data file fast thawing and reading processing:

[0142] S31. Obtaining data file information from a user query request, and obtaining a data file index matching the data file based on the data file information;

[0143] S32. Query and extract the compressed data file from the storage module according to the data file index;

[0144] S321. Obtaining a data file compression ratio from the data file information obtained in the user query request;

[0145] S322, determining an index file for the data file based on the data file compression ratio and field 1 in the index file;

[0146] S323. Determine the storage path of the data file according to field 2 in the index file;

[0147] S324. Find the storage module storing the data file according to field 2 in the index file;

[0148] S325, obtaining a data file according to field 4 in the index file;

[0149] S3251. Extract statistical information on the access frequency of the data file from the storage module, where the statistical information is the maximum and minimum values ​​of the access frequency of the data file, and calculate the median value of the access frequency based on the maximum and minimum values.

[0150] S3252: Divide the storage module into a storage area 1 having an access frequency greater than the median value of the access frequency and a storage area 2 having an access frequency less than the median value of the access frequency according to the median value of the access frequency;

[0151] S3253: Compare field 3 in the index file with the median value of the access frequency to determine the storage area;

[0152] S3254, obtain data file;

[0153] S33. Decompress the extracted compressed data file according to the data file compression ratio in the data file index.

[0154] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for quickly accessing data files, characterized in that: The following steps are involved: S1. Receive a data file and pre-process the data file, which includes the following sub-steps: S11, setting an access frequency value for the received data file; S12, extracting file information of the data file; S13. Calculate the data file compression ratio based on the file information and determine the storage method and storage path; S2. Perform quick freezing storage on the data files based on storage requirements and compression ratio, which includes the following sub-steps: S21, generating a data file index file; S22. compressing the data file according to the data file compression ratio; S23. Storing the compressed data file according to the storage method and path; S3, data file fast thawing and reading processing, including the following sub-steps: S31. Obtaining data file information from a user query request, and obtaining a data file index matching the data file based on the data file information; S32. Query and extract the compressed data file from the storage module according to the data file index; S33. Decompressing the extracted compressed data file according to the data file compression ratio in the data file index; The format of the data file index file includes the following fields: Field 1: data file compression ratio C; Field 2: storage path; Field 3: access frequency value; Field 4: data file information; The data file compression rate is calculated by using a Rapid Dynamic Compression (RDC) model, where: C=log max(N，n) min(N,n)*100%; Where C is the data file compression ratio; N is the storage limit of the storage module used to store the data file; n is the storage condition value of the data file; The storage method is determined according to the data storage requirements in the file information, and the storage path is determined according to the storage module corresponding to the data storage requirements; The storage module can dynamically sort the stored data files according to the access frequency values ​​of the data files; the order sorting rule is to sort according to the access frequency; in, In step S32, the compressed data file is searched and extracted from the storage module according to the data file index, which specifically includes the following steps: S321. Obtaining a data file compression ratio from the data file information obtained in the user query request; S322, determining an index file for the data file based on the data file compression ratio and field 1 in the index file; S323. Determine the storage path of the data file according to field 2 in the index file; S324. Find the storage module storing the data file according to field 2 in the index file; S325, obtaining a data file according to field 4 in the index file; The step S325 of obtaining the data file according to field 4 of the index file specifically includes the following steps: S3251. Extract statistical information on the access frequency of the data file from the storage module, where the statistical information is the maximum and minimum values ​​of the access frequency of the data file, and calculate the median value of the access frequency based on the maximum and minimum values. S3252: Divide the storage module into a storage area 1 having an access frequency greater than the median value of the access frequency and a storage area 2 having an access frequency less than the median value of the access frequency according to the median value of the access frequency; S3253: Compare field 3 in the index file with the median value of the access frequency to determine the storage area; S3254. Obtain data file.

2. The fast access device used in the method for fast accessing data files according to claim 1, characterized in that: The invention comprises a data receiving module (100), a hot storage pool (200), a data processing module (300), a cold storage pool (400), a thawing module (500), a quick query information storage unit (600) and an output unit (700); the data receiving module (100) is connected to a data file source via a data line, the data receiving module (100) is connected to the hot storage pool (200) and the data processing module (300) via a data bus; the hot storage pool (200) is connected to the data receiving module (100) and the data processing module (300) via a data bus; the data processing module (300) is connected to the data receiving module (100) and the data processing module (300) via a data bus; The data receiving module (100), the hot storage pool (200), the cold storage pool (400), and the quick query information storage unit (600) are connected via a data bus; the cold storage pool (400) is connected via a data bus to the data processing module (300) and the thawing module (500); the thawing module (500) is connected via a data bus to the cold storage pool (400), the hot storage pool (200), and the quick query information storage unit (600); the quick query information storage unit (600) is connected via a data bus to the data processing module (300), the thawing module (500), and the output unit (700); The output unit (700) is used to separate and obtain query conditions from the user request received by the file fast storage and reading device, and send the query conditions to the fast query information storage unit (600); then read the query results corresponding to the query conditions from the hot storage pool (200) and output them, thereby completing the fast reading of the data file; When the result data file exists in the thermal storage pool (200), the query result is "true", and the output unit (700) reads the result data file from the thermal storage pool (200) and outputs it; When the result data file does not exist in the hot storage pool (200), the query result is "false", and the output unit (700) outputs "query result does not exist"; Wherein, the data receiving module (100) comprises a stream data access unit (101) and a batch data access unit (102); The stream data access unit (101) is connected to the data processing module (300) via a data bus, and is used to provide a connection interface for the stream data file or receive the stream data file like a stream, and simultaneously identify the size, date, type, MD5 value and storage directory information of the stream data file, and transmit the accessed data file to the data processing module; wherein, the stream data access unit (101) includes a plurality of stream connectors, and the connectors are connected in parallel via a data bus; the stream connectors are created on demand by the stream data access unit (101) when the stream data file is accessed, or are created in advance and allocated on demand by the stream data access unit (101) when the stream data file is accessed; The batch data access unit (102) is connected to the hot storage pool (200) via a data bus, and is used to provide a connection interface for batch data files, access batch data files in batches, identify the size, date, type, MD5 value and storage directory information of the batch data files, and transmit the accessed data files to the hot storage pool (200); The batch data access unit (102) includes a plurality of batch connectors, which are connected in parallel via a data bus; the batch connectors are created on demand by the stream data access unit (101) when batch data files are accessed, or are created in advance and grouped according to the type of data files, and are allocated on demand by the stream data access unit (101) when batch data files are accessed.

3. The fast access device according to claim 2, wherein: The hot storage pool (200) includes a batch file storage unit (201) and a result file storage unit (202); The batch file storage unit (201) is connected to the batch data access unit (102) and the data processing module (300) via a data bus, and is used to temporarily store batch data files while waiting for processing by the data processing module (300), thereby alleviating the data processing pressure of the data processing module (300); The result file storage unit (202) is connected to the thawing module (500) and the output unit (700) via a data bus, and is used to receive and temporarily store the result data file, thereby improving the data file reading speed when the user accesses it and achieving fast reading; Wherein, the batch file storage unit (201) and the result file storage unit (202) both use magnetic media as storage media, thereby providing efficient data storage and data file access performance while taking into account costs, and improving the efficiency of data file transmission, storage and reading.

4. The fast access device according to claim 3, wherein: The data processing module (300) includes a metadata processing unit (301), a quick freezing processing unit (302) and a compression unit (303); The metadata processing unit (301) is used to extract metadata from the data file and transmit it to the fast query information storage unit (600); The quick freezing processing unit (302) is used to quick freeze the data files, that is, to classify the data files by adding data file tags and adding access rights; The compression unit (303) is used to compress and package the extracted metadata and the frozen data files based on the file directory level, achieving an operation similar to "quick freezing" the files in the directory, and then transferring the compressed and packaged data files to the cold storage pool (400) for storage, thereby reducing the file size through multi-file directory compression, thereby reducing the amount of writing to the storage medium, and ultimately achieving fast file writing; The metadata processing unit (301), the quick freezing processing unit (302) and the compression unit (303) are all connected to the data receiving module (100), the hot storage pool (200), the cold storage pool (400) and the quick query information storage unit (600) via a data bus; the data files include stream data files and batch data files, the stream data files are directly processed after being accessed by the stream data access unit (101), and the batch data files are first cached in the batch file storage unit (201) and batch processed according to a preset data volume threshold.

5. The fast access device according to claim 2, wherein: The cold storage pool (400) includes a data storage unit (401), a backup unit (402), a spare storage unit (403) and a management unit (404); The data storage unit (401) is used for long-term or permanent storage of data files processed by the data processing module (300); The backup unit (402) is used to back up the data files stored in the data storage unit (401), so that the data files can be quickly restored when the data storage unit (401) is abnormal or fails; The backup unit (403) is used to provide a backup storage resource for the data storage unit (401), so that when the data storage unit (401) fails or is damaged, the storage task and the stored data files can be quickly switched or migrated to the backup storage unit, thereby ensuring the continuity of the operation of the device and the integrity of the data storage; The management unit (404) is used to manage stored data files, and to schedule and manage the collaborative work, storage task switching and data migration among the data storage unit (401), the backup unit (402) and the backup unit (403); The data storage unit (401), the backup unit (402) and the backup storage unit (403) are all connected to the data processing module (300) and the thawing module (500) via a data bus, and all use a Blu-ray disc as a storage medium. The Blu-ray storage medium can store data files for more than 50 years, thereby realizing long-term or permanent storage of a large number of data files and reducing storage costs.

6. The fast access device according to claim 2, wherein: The thawing module (500) includes a decompression unit (501) and a thawing processing unit (502); The decompression unit (501) is used to obtain and acquire corresponding data files from the cold storage pool (400) based on the request metadata corresponding to the user query condition stored in the fast query information storage unit (600), and perform a decompression operation on the acquired data files to obtain decompressed data files; The thawing processing unit (502) is used to thaw the decompressed data file to obtain a result data file after thawing; Then, when the result data file exists in the cold storage pool (400), the result data file is uploaded to the hot storage pool (200) for user access and download; when the result data file does not exist in the cold storage pool (400), a null value is returned; The decompression unit (501) and the thawing processing unit (502) are both connected to the cold storage pool (400), the hot storage pool (200) and the fast query information storage unit (600) via a data bus.

Images