Mass small file access method and system based on inverted Sooshenwei platform

Abstract

The invention belongs to the technical field of big data processing, and particularly relates to a massive small file access method and a massive small file access system based on an inverted Sooshenwei platform. Comprising the following steps: 1, deploying a distributed file system in a SW platform server, starting a script to confirm that the file system works normally, and detecting a small file processing request; 2, judging whether a small file uploading request is received or not; if yes, skipping to the step 3; if not, ending the uploading process; 3, putting the small files into a file merging queue, and judging whether the sizes of the files in the file merging queue reach a threshold value or not; if yes, establishing inverted indexes for the small files in the queue one by one, combining all the small files into a large file, and uploading the large file to a data node of the distributed file system; if not, continuing to put the small files into the file merging queue; according to the method, the small file searching capability on the SW platform is enhanced, and small file management is facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular relates to a method and system for accessing massive small files based on an inverted index Shenwei platform. Background Art

[0002] In the wave of localization of chips, Shenwei CPU chips based on the SW64 instruction set architecture have developed rapidly and are currently widely used in servers and PCs. More and more industries and companies choose domestically developed Shenwei servers and PCs as production equipment to build information platforms. While ensuring the stable operation of business systems and architectures on Shenwei equipment, it also brings new challenges to the R&D, maintenance and iteration of business IT departments.

[0003] In the era of big data, processing large quantities of small files is a common business need in many business forms. At present, it is difficult to achieve the goal of efficient access by simply piling up server configurations and relying on a single middleware to process massive small files. At the current stage of the continuous development of the Shenwei domestic CPU ecosystem, the storage of massive small files is a blank area to be conquered.

[0004] Due to the architectural characteristics of the Shenwei platform, distributed file systems, such as HDFS, need to be customized and adapted before they can be deployed and used on the Shenwei server. In addition, the industry's distributed file systems are all oriented towards large file storage. If a large number of small files are uploaded directly, a large amount of metadata will be generated, which will consume server memory rapidly, seriously affecting the file reading speed, and may even directly bring down the entire file system.

[0005] Currently, the main way to handle small files is to merge them and store them, but simply recording the offset information of small files is too simple and crude in processing, and cannot effectively manage small files. The above merging method will cause problems such as single query method, insufficient query categories, lack of scalability, and low query efficiency. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for accessing a large number of small files based on the inverted index Shenwei platform, by merging small files into larger files and building an inverted index of small files in the process. Such an index mechanism enables the corresponding large file to be quickly located through the inverted index during the reading operation, and then the required small file is accurately intercepted according to the offset and file size information in the index. The addition of cache can significantly improve the query speed of small files.

[0007] In order to solve the above technical problems, the present invention provides a method for accessing a large number of small files based on an inverted index Shenwei platform, comprising the following steps: Step 1: Deploy a distributed file system on the Shenwei platform server, start the script to confirm that the file system is working properly, and detect small file processing requests; Step 2: Determine whether a small file upload request is received; if so, jump to step 3; if not, end the upload process; Step 3: Put the small files into the file merging queue and determine whether the file size in the file merging queue reaches the threshold; if so, create an inverted index for each of the small files in the queue, merge all the small files into a large file and upload it to the data node of the distributed file system; if not, continue to put the small files into the file merging queue; Step 4: Determine whether there are files in the queue that are blocked and waiting to be uploaded; if so, jump to step 3; if not, end the upload process; Step 5: Determine whether a small file query request is received; if so, obtain the query type and query conditions and jump to step 6; if not, end this query process; Step 6: Search the specified field in the inverted index according to the query conditions, obtain the index records that meet the query conditions, and obtain the complete small file information; Step 7: Determine whether the small file exists in the cache; if so, directly obtain it from the cache and end this query process; if not, jump to step 8; Step 8: Obtain the storage information of the small file in the distributed file system from the index record of the inverted index, and parse the specified query small file according to the storage offset in the file and the small file size information; Step 9: Determine whether the number of accesses to the small file within the set cycle time reaches a threshold; if so, add the queried small file to the cache and end the query process; if not, directly end the query process.

[0008] Preferably, in step one, the distributed file system is run through the Shenwei platform server, and the distributed file system adopts the version of the Shenwei platform, and the script used is modified for compatibility to a customized version through the Shenwei platform.

[0009] Preferably, in step three, the merge queue setting threshold should be smaller than the size of the storage unit set on the distributed file system.

[0010] Preferably, in step three, the index data of the inverted index includes at least five fields: the name of the small file, the file size, the file name of the corresponding merged file stored on the file system, the offset in the merged file stored on the file system, and the file upload time; the remaining file-related information supports user-defined extensions; each field records the specific content information, inverted index and attribute information structure of the file corresponding to the field.

[0011] Preferably, the complete index structure of each field includes an inverted list and a dictionary; wherein the inverted list is an ordered array that stores all document IDs matching a certain word and the position information of the word appearing in the document; the dictionary is a string set consisting of all words that have appeared in a document collection, and each index item in the dictionary records the word information itself and a pointer to the inverted list.

[0012] Preferably, in step 5, the query type includes: Exact query is used to make equal value judgment on the query content. The query content is equal to the value of the corresponding field in the file attribute; Fuzzy query, used to support querying data without fully determining the search conditions, requires the use of wildcards to represent uncertain characters or character sequences; Range query is used to set an interval and query the data that falls within the interval. It supports numbers and dates that are easy to compare.

[0013] Preferably, in step six, the specific method for searching the specified field includes: finding the index record of the matching item in the inverted index according to the query content and the inverted index, obtaining the corresponding document ID in the index record, and obtaining the specific information of the small file from other fields according to the document ID, and the specific information includes the storage location information in the file system.

[0014] Preferably, after directly obtaining from the cache in step seven, it also includes: determining whether the time since the corresponding small file in the cache was last accessed has reached a clearing time threshold; if so, deleting the corresponding small file from the cache; wherein the clearing time threshold can be configured by itself and is the access time interval period of the small file.

[0015] Preferably, expired cache files are cleared by scanning the cache with a scheduled script. The specific method of clearing includes: scanning one by one with a cache deletion script at a set time interval and determining whether the corresponding small files in the cache have reached a clearing time threshold since they were last accessed.

[0016] The present invention also provides a massive small file access system based on the inverted index Shenwei platform, which adopts a massive small file access method based on the inverted index Shenwei platform as described above, including a file upload function module and a file query function module; wherein the file upload function module includes a file request processing module, a file merging module, an index module and a file upload module; the file query function module includes a file request processing module, an index module, a cache module and a file acquisition module; The file request processing module is responsible for processing requests from users, including file upload and file query requests; The file merging module is responsible for merging the incoming small files; The index module is responsible for creating an inverted index for each small file and recording file attribute information and storage information; The file upload module is responsible for uploading the merged file; The cache module is responsible for determining whether to cache corresponding small files or clear the cache according to the user query situation; The file acquisition module is responsible for acquiring the queried small files from the file system based on the inverted index result.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention optimizes the data storage and query process by the following steps: merging small files into larger files, and constructing an inverted index of small files in this process. Such an indexing mechanism enables the corresponding large file to be quickly located through the inverted index during the read operation, and then the required small file is accurately intercepted according to the offset and file size information in the index, and the addition of the cache can significantly improve the query speed of small files. The existence of the inverted index also enriches the way to query files, supports fuzzy query, precise query, range query and even complex combination condition query, etc., greatly enhances the search ability for small files on the Shenwei platform, and helps small file management. In addition, this method also completes the breakthrough of the Shenwei platform in accessing massive small files. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the inverted index structure diagram corresponding to the small file of the present invention.

[0019] Figure 2 This is the main flow chart of small file upload and storage of the present invention.

[0020] Figure 3 This is the main flow chart of small file query of the present invention.

[0021] Figure 4 The structure diagram of the access system applied by the method of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in very simplified form and in non-precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0023] The following Table 1 is a schematic diagram of the specific data information of the small file to be indexed information in an embodiment of the present invention, which has five fields: file name name, file size size, offset stored in the distributed file system offset, merged file name mergename stored in the file system, and file creation time starttime.

[0024] Table 1: field name size offset mergename starttime File to be indexed information Pomeranian puppy picture 1 1300 2200 mergefile1 2023-10-115:27:33 Table 2 below shows the word segmentation items of the file names Pomeranian puppy picture one and catpicture two. In this way, when searching for keywords such as Pomeranian, puppy, and picture, the document ID corresponding to the file can be matched to obtain the specific information of the file.

[0025] Table 2: file name Participle Term Pomeranian puppy picture 1 Pomeranian, puppy, dog, picture, one, Pomeranian puppy, puppy picture, picture, puppy picture one, Pomeranian puppy picture one Catpicture2 catpicuturetwo,two,picture,cat,catpicture like Figure 1 The figure shows the inverted index structure corresponding to the small file. The complete index structure of each field includes an inverted table and a dictionary. The inverted table is an ordered array that stores all document IDs that match a word and the location information of the word in the document. The dictionary is a string set consisting of all words that have appeared in the document collection. Each index item in the dictionary records the word information itself and a pointer to the inverted table.

[0026] like Figure 2 As shown, the embodiment of the present invention provides a main process for uploading and storing small files, which specifically includes the following steps: a1: Determine whether a file upload request is received. If so, proceed to step a2; otherwise, end the process.

[0027] a2: Add the files to be uploaded to the merge queue.

[0028] a3: Determine whether the length of the merge queue has reached a threshold. The time to trigger the judgment is when a file is added to the merge queue. If it is determined to have reached the threshold, the file is put into the queue and blocked for waiting. All the files in the queue are taken out and enter step a4.

[0029] a4: Merge the files to be uploaded in the merge queue.

[0030] a5: Create an inverted index for each file to be merged. The index information includes five fields: the name and size of the small file, the file name of the corresponding merged file stored on the file system, the offset in the merged file stored on the file system, and the file upload time. The remaining file-related information such as file type supports user-defined extensions. These attribute information metadata and index data are stored on the Shenwei server that manages index data.

[0031] a6: Store the merged file on at least one data node of the distributed file system. Whether to back up the file can be set based on timing needs.

[0032] a7: Determine whether there is a file in the queue waiting to be uploaded. If so, jump back to step a2 to continue the process; otherwise, end the upload process.

[0033] like Figure 3 As shown, the embodiment of the present invention provides a main process of small file query, which specifically includes the following steps: b1: Determine whether a file query request is received. If so, jump to step b2; otherwise, end the process.

[0034] b2: Determine the file query type, precise query, fuzzy query or range query. Precise query is to make an equivalence judgment on the query content, and the query content is equal to the value of the corresponding field in the file attributes; fuzzy query supports querying data when the search conditions are not completely determined, but wildcards are required to represent uncertain characters or character sequences. Commonly used wildcards include "%" for character sequences of any length, "_" for any single character, etc.; range query is to formulate an interval query to find data that meets the range of the interval, such as >=1 and <3, etc., and generally supports field types such as numbers and dates that are easy to compare.

[0035] b3: Parse the query request content to obtain the query field and query conditions.

[0036] b4: Search the query field specified by the inverted index, and find the document ID that meets the query conditions based on the mapping relationship between the query content's corresponding metadata, dictionary, and inverted index.

[0037] b5: A small file is uniquely identified by the document ID. The document ID is the unique key in the index. The document ID can be used to lock other fields of the file storage information in the inverted index, obtain all field information, and thus obtain complete small file information, including the name and size of the small file, the file name of the corresponding merged file stored on the file system, the offset in the merged file stored on the file system, the file upload time, etc.

[0038] b6: Determine whether the small file already exists in the cache. If so, directly obtain the small file and the current process ends. Otherwise, jump to step b7.

[0039] In actual use, some files are likely to be accessed multiple times, so establishing a cache is very effective in improving access efficiency. To improve cache performance, the cache part is placed on a separate Shenwei server and isolated from the index server, which can support efficient reading under multiple concurrency. Given the different access frequencies of each file, in order to further improve reading efficiency, each time a user reads a file, an access record will be recorded to count the number of accesses. Files with high access frequencies will be stored in the cache server. When the file is read again, it only needs to be directly obtained from the cache, which can greatly reduce access time.

[0040] b7: Read the merged file from the distributed file system, and parse the specified query small file according to the storage offset and small file size information.

[0041] b8: Determine whether the number of accesses to the corresponding small file within a set period of time reaches a threshold, such as setting the access threshold to 8 within 12 hours; if the threshold is reached within the period of time, jump to step b9.

[0042] The specific judgment logic is to subtract 12 hours from the most recent access time, obtain the number of access records of small files accessed during this time period, that is, the number of accesses, and determine whether the number of accesses reaches the threshold.

[0043] b9: Add the found small files to the cache and end the process.

[0044] Among them, the clearing of expired cache files is carried out by scanning the cache through a scheduled script. The specific method is: at the set time interval, use the cache deletion script to scan one by one and determine whether the corresponding small file in the cache has reached the clearing time threshold since the last access time. The clearing time threshold is the access time interval period of the small file, which can be configured by yourself, such as 24 hours; if the threshold is reached and there is no access record of the small file in the last 24 hours, the corresponding small file will be deleted from the cache.

[0045] like Figure 4 As shown, the access system used by the method of the present invention mainly includes file upload and file query function modules, wherein the file upload function module includes a file request processing module, a file merging module, an index module, and a file upload module; the file query function module includes a file request processing module, an index module, a cache module, and a file acquisition module. The specific functions of each module are as follows: The file request processing module is responsible for processing requests from users, including file uploads, file query requests, etc. The file merging module is responsible for merging incoming small files; The index module is responsible for creating an inverted index for each small file and recording file attribute information and storage information; File upload module, responsible for uploading the merged file; The cache module is responsible for determining whether to cache corresponding small files or clear the cache based on user query conditions; The file acquisition module is responsible for acquiring the queried small files from the file system based on the inverted index results.

[0046] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A method for accessing massive small files based on the inverted index Shenwei platform, characterized in that: The steps include: Step 1: Deploy a distributed file system on the Shenwei platform server, start the script to confirm that the file system is working properly, and detect small file processing requests; Step 2: Determine whether a small file upload request is received; if so, jump to step 3; if not, end the upload process; Step 3: Put the small files into the file merging queue and determine whether the file size in the file merging queue reaches the threshold; if so, create an inverted index for each of the small files in the queue, merge all the small files into a large file and upload it to the data node of the distributed file system; if not, continue to put the small files into the file merging queue; Step 4: Determine whether there are files in the queue that are blocked and waiting to be uploaded; if so, jump to step 3; if not, end the upload process; Step 5: Determine whether a small file query request is received; if so, obtain the query type and query conditions and jump to step 6; if not, end this query process; Step 6: Search the specified field in the inverted index according to the query conditions, obtain the index records that meet the query conditions, and obtain the complete small file information; Step 7: Determine whether the small file exists in the cache; if so, obtain it directly from the cache and end the query process; if not, jump to step 8; Step 8: Obtain the storage information of the small file in the distributed file system from the index record of the inverted index, and parse the specified query small file according to the storage offset in the file and the small file size information; Step 9: Determine whether the number of accesses to the small file within the set cycle time reaches a threshold; if so, add the queried small file to the cache and end the query process; if not, directly end the query process.

2. A method for accessing a large number of small files based on the inverted index Shenwei platform as claimed in claim 1, characterized in that: In the step 1, a distributed file system is run through the Shenwei platform server, and the distributed file system adopts the version of the Shenwei platform. The script used is modified for compatibility through the Shenwei platform to a customized version.

3. A method for accessing a large number of small files based on the inverted index Shenwei platform as claimed in claim 1, characterized in that: In the step 3, the merge queue setting threshold should be smaller than the size of the storage unit set on the distributed file system.

4. A method for accessing a large number of small files based on the inverted index Shenwei platform as claimed in claim 1, characterized in that: In the step three, the index data of the inverted index includes at least five fields: the name of the small file, the file size, the file name of the corresponding merged file stored on the file system, the offset in the merged file stored on the file system, and the file upload time; the remaining file-related information supports user-defined extension; each field records the specific content information, inverted index and attribute information structure of the file corresponding to the field.

5. A method for accessing a large number of small files based on the inverted index Shenwei platform as claimed in claim 1, characterized in that: The complete index structure of each field includes an inverted list and a dictionary; wherein the inverted list is an ordered array that stores all document IDs that match a certain word and the position information of the word in the document; the dictionary is a string set consisting of all words that have appeared in the document collection, and each index item in the dictionary records the word information itself and a pointer to the inverted list.

6. A method for accessing a large number of small files based on the inverted index Shenwei platform as claimed in claim 1, characterized in that: In step 5, the query type includes: Exact query is used to make equal value judgment on the query content. The query content is equal to the value of the corresponding field in the file attribute. Fuzzy query, used to support querying data without fully determining the search conditions, requires the use of wildcards to represent uncertain characters or character sequences; Range query is used to set an interval and query the data that falls within the interval. It supports numbers and dates that are easy to compare.

7. A method for accessing a large number of small files based on the inverted index Shenwei platform as claimed in claim 1, characterized in that: In step six, the specific method for searching the specified field includes: finding the index record of the matching item in the inverted index according to the query content and the inverted index, obtaining the corresponding document ID in the index record, and obtaining the specific information of the small file from other fields according to the document ID, and the specific information includes the storage location information in the file system.

8. A method for accessing a large number of small files based on the inverted index Shenwei platform as claimed in claim 1, characterized in that: After directly obtaining from the cache in step seven, the step also includes: determining whether the time from the last access time of the corresponding small file in the cache reaches a clearing time threshold; if so, deleting the corresponding small file from the cache; wherein the clearing time threshold can be configured by itself and is the access time interval period of the small file.

9. A method for accessing a large number of small files based on the inverted index Shenwei platform as claimed in claim 8, characterized in that: Expired cache files are cleared by scanning the cache with a scheduled script. The specific method of clearing includes: scanning one by one with a cache deletion script at a set time interval and determining whether the corresponding small files in the cache have reached a clearing time threshold since they were last accessed.

10. A massive small file access system based on the inverted index Shenwei platform, using a massive small file access method based on the inverted index Shenwei platform as described in any one of claims 1 to 9, characterized in that: It includes a file upload function module and a file query function module; wherein the file upload function module includes a file request processing module, a file merging module, an index module and a file upload module; the file query function module includes a file request processing module, an index module, a cache module and a file acquisition module; The file request processing module is responsible for processing requests from users, including file upload and file query requests; The file merging module is responsible for merging the incoming small files; The index module is responsible for creating an inverted index for each small file and recording file attribute information and storage information; The file upload module is responsible for uploading the merged file; The cache module is responsible for determining whether to cache corresponding small files or clear the cache according to the user query situation; The file acquisition module is responsible for acquiring the queried small files from the file system based on the inverted index result.

Images