Mass small file storage and real-time retrieval method and system based on Elasticsearch

Abstract

The invention provides a massive small file storage and real-time retrieval method and system based on Elasticsearch, which are used for solving the problems that the existing massive small file storage scheme cannot solve the problems of classification matching, real-time retrieval and the like of massive small files, or the offline deployment and maintenance cost is high and efficient bottom query interface support is lacked during retrieval. Or small file classification and query scenes need to be analyzed in advance, and certain universality is lacked. According to the Elasticsearch-based massive small file storage and real-time retrieval method provided by the invention, by adding boundary processing of the small files, metadata information and index information of the massive small files and file data are separately stored, and a plurality of small files are combined into a large file in a file storage system; therefore, the file storage system not only can efficiently store massive small files, but also supports massive storage of large files, and the compatibility of file storage is effectively enhanced.

Description

Technical Field

[0001] This invention relates to file storage and retrieval methods, and more particularly to a method and system for storing and retrieving massive amounts of small files in real time based on Elasticsearch. Background Technology

[0002] File storage typically requires recording metadata information such as the file's type, size, storage location, offset, and permissions. When the number of files is small, this metadata is usually stored in memory for fast storage and retrieval. However, when the file size reaches millions or even tens of millions, this metadata will exhaust server memory, causing service unavailability. This is precisely why the HDFS big data storage component cannot directly support the storage of massive numbers of small files. Therefore, the core solution to the problem of storing and retrieving massive numbers of small files lies in the processing of their metadata information.

[0003] Currently, the main solutions for handling the storage of massive amounts of small files are as follows:

[0004] (I) HDFS Small File Storage. HDFS boasts numerous excellent features such as high fault tolerance, scalability, and low storage costs. While it cannot directly support real-time storage of massive amounts of small files, it can use MapReduce to batch merge small files offline, storing them as large files in formats such as SequenceFile and MapFile, thereby reducing the memory footprint of metadata on the Master node. This approach effectively manages and optimizes the storage of massive amounts of small files in a big data environment while ensuring scalability. Its replication strategy also enables high availability. However, it does not address issues such as the classification and matching of massive amounts of small files and real-time retrieval.

[0005] (ii) Using cloud storage and distributed object storage. These storage systems are relatively mature, such as Ceph and FastDFS. They generally exist in the form of distributed file systems and can well support the storage of massive amounts of small files. However, the offline deployment and maintenance costs of this solution will increase significantly, and for file retrieval scenarios, it often lacks efficient underlying query interface support.

[0006] (III) Using NoSQL Databases. For small file storage scenarios requiring frequent access, NoSQL databases can be used. They can satisfy the storage and querying of large amounts of structured metadata information, while providing excellent random read / write and distributed scalability capabilities. However, this solution requires prior analysis of small file classification and query scenarios, and customized design of storage rowkeys that meet the query scenarios. Otherwise, query performance will significantly decrease. In addition, this solution does not support SQL query standards and is only suitable for specific scenarios involving massive small file processing, lacking a certain degree of versatility. Summary of the Invention

[0007] The purpose of this invention is to solve the problems of existing massive small file storage and processing solutions, such as failure to solve the problems of classification and matching of massive small files and real-time retrieval, or high offline deployment and maintenance costs, lack of efficient underlying query interface support during retrieval, or the need to analyze the small file classification and query scenarios in advance, and lack of certain general technical problems. The invention provides a method and system for massive small file storage and real-time retrieval based on Elasticsearch.

[0008] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0009] A method for storing and retrieving massive amounts of small files in real time based on Elasticsearch, which is unique in that it includes both file storage and file retrieval methods;

[0010] The file storage method includes the following steps:

[0011] Step 1: Collect the initial metadata information of the file and determine whether the current file size is less than the maximum threshold for small files. If so, proceed to Step 2; otherwise, mark the write mode as new and proceed directly to Step 4.

[0012] Step 2: Check if there is a key in the metadata cache module that is smaller than the maximum threshold for small files. If it exists, retrieve the corresponding metadata information from Elasticsearch based on the key, and update the offset of the retrieved metadata information by adding the current file size. Determine if the updated offset exceeds the maximum threshold for small files. If it does, proceed to Step 3. Otherwise, update the starting position, offset, and verification code corresponding to the key, mark the write method as append, and proceed to Step 4. If it does not exist, proceed to Step 3.

[0013] Step 3: Generate a new storage path and use the characters generated according to the business rules as the key corresponding to the storage path. Mark the write method as new and set the initial offset to zero. Update the metadata cache module with the storage path, offset, write method, and write date as the value corresponding to the key and cache them. Then write the updated metadata information to Elasticsearch.

[0014] Step 4: The metadata information of the current file is handed over to the meta-index processing module to generate valid index information and then written to Elasticsearch; then the file data of the current file is written to or appended to the file storage system through the data writer.

[0015] Step 5: Repeat steps 1 through 4 until all files have been saved.

[0016] The document retrieval method includes the following steps:

[0017] Step 1: During file retrieval, query conditions are formed based on the information to be retrieved, and the query conditions are sent to the search engine. The search engine processes the query conditions into an expression acceptable to Elasticsearch and sends a query request to Elasticsearch.

[0018] Step 2: The search engine obtains the file metadata information that matches the query conditions, and obtains the actual file data stream through the data reader;

[0019] Step 3: The search engine returns the obtained file data stream and file metadata information to the client.

[0020] Furthermore, in step 1 of the file storage method, the initial metadata information of the file includes the original file name, file size, file type, checksum, creator, creation time, and permissions.

[0021] Furthermore, in steps 1 and 2 of the file storage method, the maximum threshold for small files is 128M.

[0022] Furthermore, in step 3 of the file storage method, a new storage path is generated according to the format root / yyyy / mm / uuid, where root represents the root directory, yyyy represents the year, mm represents the month, and uuid represents the generated character.

[0023] In addition, the present invention also provides a massive small file storage and real-time retrieval system based on Elasticsearch, which is used to implement the above-mentioned massive small file storage and real-time retrieval method based on Elasticsearch. Its special feature is that it includes a preprocessing module, a classification module, a metadata processing module, a meta-index processing module, a data storage module, a data reader, a search engine module, and an Elasticsearch module.

[0024] The preprocessing module is used to collect the original metadata information of the file and preprocess it;

[0025] The input of the classification module is connected to the output of the preprocessing module, and is used to mark the preprocessed file as a large file or a small file according to the file size;

[0026] The input end of the metadata processing module is connected to the first output end of the classification module, and is used to determine the file writing method and storage path according to the file size;

[0027] The two inputs of the meta-index processing module are respectively connected to the second output of the classification module and the first output of the metadata processing module, and are used to catalog the metadata information of the file to generate valid index data.

[0028] The first input of the Elasticsearch module is connected to the second output of the metadata processing module, which is used to update and write all metadata information of the file into Elasticsearch. Its second input is connected to the second output of the meta-index processing module, which is used to update and write index data into Elasticsearch.

[0029] The data storage module includes a data writer and a file storage system connected in sequence. The input end of the data writer is connected to the first output end of the meta-index processing module, which is used to safely and efficiently write file data into the file storage system for storage.

[0030] The data reader communicates bidirectionally with both the file storage system and the search engine module. The search engine module also communicates bidirectionally with the Elasticsearch module. The search engine module is used to input standardized query conditions and retrieve the corresponding file metadata information from Elasticsearch based on the query conditions. The data reader is used to analyze the storage path, start position, and offset based on the file metadata information retrieved by the search engine module, and then read the corresponding file data from the file storage system based on the storage path, start position, and offset. Finally, the search engine module returns the read file data to the client.

[0031] Furthermore, the file storage system employs HDFS or distributed object storage.

[0032] Furthermore, the data writer adopts a facade design, abstracting the file append and write operations of the file storage system into a standard interface.

[0033] Furthermore, the data reader adopts a facade design to abstract the file reading function of the file storage system.

[0034] The advantages of this invention compared to the prior art are as follows:

[0035] 1. This invention provides a method for storing and retrieving massive small files in real time based on Elasticsearch. It innovatively separates the metadata and index information of massive small files from the file data by adding boundary processing for small files. Furthermore, it merges multiple small files into a large file within the file storage system. This allows the file storage system to efficiently store massive small files while also supporting massive storage of large files, effectively enhancing the compatibility of file storage. It overcomes the inherent limitations of NoSQL and FastDFS file storage systems in storing massive large files. Since large and small files typically coexist in real-world scenarios, this invention is more practical in practical applications.

[0036] 2. This invention provides a method for storing and retrieving massive small files in real time based on Elasticsearch. By leveraging the high-performance full-text search service provided by Elasticsearch, the retrieval of massive small files becomes faster and more reliable. Through its scalable data storage and vector search capabilities, it provides more possibilities for complex query scenarios, such as "image search" and intelligent search of cloud storage, greatly improving query performance and effectiveness, and making its applicable scenarios more diversified.

[0037] 3. The present invention provides a massive small file storage and real-time retrieval system based on Elasticsearch. Through Elasticsearch, it can not only support the retrieval of file metadata information, but also support other complex retrieval scenarios. Utilizing its vector search function, it can effectively support the retrieval and processing of various natural languages, reduce the dependence on other traditional hardware devices, and reduce the cost of system deployment and maintenance. Attached Figure Description

[0038] Figure 1 This is a schematic diagram illustrating the principle of a method for storing and retrieving massive amounts of small files in real time based on Elasticsearch, as described in this invention. Detailed Implementation

[0039] To make the advantages and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Elasticsearch is an open-source search and analytics engine built on Apache Lucene, designed for cloud computing and capable of rapidly processing large amounts of data. It provides a distributed, multi-user, full-text search engine based on a RESTful web interface. Developed by Shay Banon in 2010, it is open-source. Key features and functionalities of Elasticsearch include:

[0041] 1. Full-text search: Elasticsearch provides high-performance full-text search capabilities, supports complex queries such as fuzzy search and phrase search, and supports highlighting.

[0042] 2. Data Analysis: In addition to search, Elasticsearch also has powerful data analysis capabilities, supporting various scenarios such as log analysis and real-time monitoring.

[0043] 3. Real-time performance: Elasticsearch can process data in real time, providing near real-time search and analysis capabilities, ensuring that data updates are immediately visible.

[0044] 4. Scalability: Elasticsearch is distributed and can be easily scaled to multiple nodes to handle large datasets and high concurrency requests.

[0045] 5. Natural Language Processing (NLP) Support: Elasticsearch 8 adds native support for natural language processing models. Users can directly use NLP models for inference in Elasticsearch without additional components or coding to achieve functions such as named entity recognition, sentiment analysis, and text classification.

[0046] 6. Vector Search Functionality: Elasticsearch 8 enhances vector search functionality, supporting high-speed queries on large-scale data.

[0047] like Figure 1 As shown, a method for storing and retrieving massive amounts of small files in real time based on Elasticsearch includes a file storage method and a file retrieval method. The file storage method includes the following steps:

[0048] I. Document Information Collection.

[0049] Collect the original metadata information of the file. The original metadata information of the file mainly includes, but is not limited to, the original file name, file size, file type, checksum, creator, creation time, permissions, etc.

[0050] In this invention, large files and small files in a file are processed separately. Large files are stored directly, while small files are processed by adding boundary conditions to separate their metadata and index information from the file data. In the file storage system, multiple small files are merged and stored as a large file.

[0051] First, a threshold is set to determine whether a file is a small file. This threshold is denoted as the maximum threshold for small files. It can be determined based on the characteristics of the storage system. A reasonable threshold can significantly reduce storage space usage; in this embodiment, it is set to 128MB. After the original metadata information of the file is collected, the file size in the file metadata information is compared with the maximum threshold for small files. If the file size is less than the maximum threshold, it proves that the current file is a small file, and step 2 is executed. If the file size is greater than or equal to the maximum threshold, it proves that the current file is a large file, and the write method is marked as "new and then stored directly".

[0052] II. Metadata processing for small files.

[0053] First, a custom metadata caching module is defined. Each time the storage path of a small file is determined, the module is checked to see if a key smaller than the maximum threshold for small files exists. If no key exists, the module is either in its initialization state or the current storage path's capacity is full, necessitating the generation of a new storage path. This invention generates storage paths in the format root / yyyy / mm / uuid, generating characters according to business rules and using these characters as the key for the current storage path. The write method is marked as "new," with an initial offset of zero. Here, root represents the root directory, yyyy represents the year, mm represents the month, and uuid represents the generated characters. The storage path, offset, write method, and write date are then updated as the value corresponding to the key in the metadata caching module for caching. The current storage path, offset, write method, and write date are also updated in the current file's metadata information, and the updated metadata information is written to Elasticsearch.

[0054] If a key smaller than the maximum threshold for small files exists in the metadata cache module, it indicates that the location corresponding to the current storage path in the metadata cache module has capacity. In this case, the corresponding metadata information is retrieved from Elasticsearch based on the key, and the file size of the current file is added to the offset of the retrieved metadata information. The updated offset is then used to determine whether the storage capacity of the current storage path exceeds the maximum threshold for small files. If so, it indicates that the storage path is insufficient, and a new storage path needs to be generated using the above method. Otherwise, the file is stored through the current storage path, and the starting position, offset data, and verification code corresponding to the key are updated. The write method is marked as append before being updated in Elasticsearch. At this point, the metadata information of the file in Elasticsearch includes the original filename, file type, file size, starting position, offset, storage path, write method, verification code, write date, creator, creation time, permissions, etc., which facilitates subsequent file retrieval.

[0055] Third, the updated metadata information of the current file is handed over to the meta-index processing module to generate valid index information, which is then written to Elasticsearch. Next, the data writer writes or appends the file data of the current file to the file storage system, thus completing the storage of a file. The checksum in the file metadata information ensures data integrity during storage and retrieval. The file storage system supports the use of HDFS (Hadoop Distributed File System) or distributed object storage to meet high availability and high scalability requirements.

[0056] 4. Repeat steps one through three to complete the storage of all files.

[0057] The document retrieval method includes the following steps:

[0058] When retrieving documents, query conditions are formed based on the information to be retrieved, and these query conditions are sent to the search engine. The search engine processes the query conditions into an expression acceptable to Elasticsearch and sends a query request to Elasticsearch.

[0059] The search engine retrieves the file metadata information that matches the query criteria and uses a data reader to obtain the actual file data stream. Finally, the search engine returns the retrieved file data stream and file metadata information to the client, completing the file retrieval.

[0060] The method for storing and retrieving massive small files based on Elasticsearch provided by this invention can efficiently store massive small files as well as massive large files, meeting the requirements of high availability and high scalability, while also enabling fast and real-time classification, matching and retrieval of these small files.

[0061] To implement the above-mentioned method for storing and retrieving massive small files in real time based on Elasticsearch, this invention provides a system for storing and retrieving massive small files in real time based on Elasticsearch, including a preprocessing module, a metadata processing module, a meta-index processing module, a data storage module, a data reader, a search engine module, and an Elasticsearch module.

[0062] The preprocessing module is used to collect the original metadata information of the uploaded files and preprocess it. The preprocessing includes small file checks, file permission checks, generating new rule filenames, and standardizing the metadata information format.

[0063] The input of the metadata processing module is connected to the first output of the preprocessing module. It receives the preprocessed file and determines the file's write method and storage path based on its size. Specifically, the preprocessed file is categorized into large or small files based on its size, and stored according to the respective write method and storage path. For large files, metadata indexing is performed directly; for small files, the metadata is appended to the current storage path. The first output of the metadata processing module directly outputs the large file, and its second output outputs the processed small file.

[0064] The metadata processing module is responsible for combining business system rules. Its two inputs are connected to the second and third outputs of the metadata processing module, respectively, to catalog the metadata information of large or small files and generate valid index data.

[0065] The first input of the Elasticsearch module is connected to the second output of the metadata processing module to update the metadata information of files in Elasticsearch. Its second input is connected to the second output of the meta-index processing module to update the index data in Elasticsearch.

[0066] The data storage module comprises a data writer and a file storage system connected in sequence. The data writer adopts a facade design, with its input end connected to the first output end of the meta-index processing module. It is used to abstract the file append and write operations of the underlying file storage system into a standard interface, responsible for securely and efficiently writing file data to the file storage system for later use. The underlying file storage system can be configured according to actual needs based on factors such as data structure, access speed, scalability, ease of maintenance, and cost, for example, choosing HDFS or distributed object storage (Ceph or others).

[0067] The data reader communicates bidirectionally with both the file storage system and the search engine module. The search engine module also communicates bidirectionally with the Elasticsearch module. The search engine module takes standardized query conditions as input and retrieves the corresponding file metadata information from Elasticsearch based on these conditions. The data reader analyzes the storage path, start position, and offset based on the file metadata information retrieved by the search engine module, and then reads the corresponding file data from the file storage system based on these conditions. Finally, the search engine module returns the read file data to the client.

[0068] The search engine module primarily encapsulates various API operations for interacting with the Elasticsearch module. The data reader also employs a facade design, abstracting the file reading functionality of the file storage system to facilitate adaptation to different file storage systems.

[0069] This invention pertains to a general technology in the field of file processing and can be widely applied to scenarios including but not limited to knowledge bases, smart cloud storage, and big data file processing. This technology has previously been used in big data cloud storage platforms.

[0070] The above description is only used to illustrate the technical solutions of the present invention, and is not intended to limit them. For those skilled in the art, modifications can be made to the specific technical solutions described in the above embodiments, or equivalent substitutions can be made to some of the technical features. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions protected by the present invention.

Claims

1. A method for storing and retrieving massive amounts of small files in real time based on Elasticsearch, characterized in that, This includes file storage methods and file retrieval methods; The file storage method includes the following steps: Step 1: Collect the initial metadata information of the file and determine whether the current file size is less than the maximum threshold for small files. If so, proceed to Step 2; otherwise, mark the write mode as new and proceed directly to Step 4. Step 2: Check if there is a key in the metadata cache module that is smaller than the maximum threshold for small files. If it exists, retrieve the corresponding metadata information from Elasticsearch based on the key, and update the offset of the retrieved metadata information by adding the current file size. Determine if the updated offset exceeds the maximum threshold for small files. If it does, proceed to Step 3. Otherwise, update the starting position, offset, and verification code corresponding to the key, mark the write method as append, and proceed to Step 4. If it does not exist, proceed to Step 3. Step 3: Generate a new storage path and use the characters generated according to the business rules as the key corresponding to the storage path. Mark the write method as new and set the initial offset to zero. Update the metadata cache module with the storage path, offset, write method, and write date as the value corresponding to the key and cache them. Then write the updated metadata information to Elasticsearch. Step 4: The metadata information of the current file is handed over to the meta-index processing module to generate valid index information and then written to Elasticsearch; then the file data of the current file is written to or appended to the file storage system through the data writer. Step 5: Repeat steps 1 through 4 until all files have been saved. The document retrieval method includes the following steps: Step 1: Based on the information to be retrieved, form query conditions and send the query conditions to the search engine. The search engine processes the query conditions into an expression acceptable to Elasticsearch and sends a query request to Elasticsearch. Step 2: The search engine obtains the file metadata information that matches the query conditions, and obtains the actual file data stream through the data reader; Step 3: The search engine returns the obtained file data stream and file metadata information to the client.

2. The method for storing and retrieving massive amounts of small files in real time based on Elasticsearch according to claim 1, characterized in that: In step 1 of the file storage method, the initial metadata information of the file includes the original file name, file size, file type, checksum, creator, creation time, and permissions.

3. The method for storing and retrieving massive amounts of small files in real time based on Elasticsearch according to claim 2, characterized in that: In steps 1 and 2 of the file storage method, the maximum threshold for small files is 128M.

4. The method for storing and retrieving massive amounts of small files in real time based on Elasticsearch according to claim 3, characterized in that: In step 3 of the file storage method, a new storage path is generated according to the format root / yyyy / mm / uuid, where root represents the root directory, yyyy represents the year, mm represents the month, and uuid represents the generated character.

5. A massive small file storage and real-time retrieval system based on Elasticsearch, used to implement the massive small file storage and real-time retrieval method based on Elasticsearch as described in any one of claims 1-4, characterized in that: It includes a preprocessing module, a classification module, a metadata processing module, a meta-index processing module, a data storage module, a data reader, a search engine module, and an Elasticsearch module; The preprocessing module is used to collect the original metadata information of the file and preprocess it; The input of the classification module is connected to the output of the preprocessing module, and is used to mark the preprocessed file as a large file or a small file according to the file size; The input end of the metadata processing module is connected to the first output end of the classification module, and is used to determine the file writing method and storage path according to the file size; The two inputs of the meta-index processing module are respectively connected to the second output of the classification module and the first output of the metadata processing module, and are used to catalog the metadata information of the file to generate valid index data. The first input of the Elasticsearch module is connected to the second output of the metadata processing module, which is used to update and write all metadata information of the file into Elasticsearch. Its second input is connected to the second output of the meta-index processing module, which is used to update and write index data into Elasticsearch. The data storage module includes a data writer and a file storage system connected in sequence. The input end of the data writer is connected to the first output end of the meta-index processing module, and is used to write file data into the file storage system for storage. The data reader communicates bidirectionally with both the file storage system and the search engine module. The search engine module also communicates bidirectionally with the Elasticsearch module. The search engine module is used to input standardized query conditions and retrieve the corresponding file metadata information from Elasticsearch based on the query conditions. The data reader is used to analyze the storage path, start position, and offset based on the file metadata information retrieved by the search engine module, and then read the corresponding file data from the file storage system based on the storage path, start position, and offset. Finally, the search engine module returns the read file data to the client.

6. The massive small file storage and real-time retrieval system based on Elasticsearch according to claim 5, characterized in that: The file storage system uses HDFS or distributed object storage.

7. The massive small file storage and real-time retrieval system based on Elasticsearch according to claim 6, characterized in that: The data writer adopts a facade design, which abstracts the file append and write operations of the file storage system into a standard interface.

8. The massive small file storage and real-time retrieval system based on Elasticsearch according to claim 7, characterized in that: The data reader adopts a facade design to abstract the file reading function of the file storage system.

Images