Method, System, Device and Medium for Accelerating Local Full-Text Retrieval of ES Files

Abstract

The present invention discloses a method, system, device and medium for accelerating local full-text retrieval of ES files. The method includes the following steps: collecting file information of folders in the disk by using the file record reading function of the fsutil tool; saving the file information to a preset program memory and naming it original; obtaining the full path of the file in the disk according to original, processing the full path to obtain a file list; uploading the file content in the disk to the Elasticsearch database according to the file list; inputting the keyword to be retrieved, and obtaining the retrieval result according to the keyword and the Elasticsearch database. The present invention uses fsutil to view usn-related information and initializes it as a memory file path mapping Map, finds out all local file paths and extracts the content and stores it in the Elasticsearch database, and can directly retrieve during full-text retrieval of Elasticsearch-related APIs; greatly improves the efficiency of data upload and can be widely applied to the field of data retrieval technology.

Description

Technical Field

[0001] The present invention relates to the technical field of data retrieval, and in particular to a method, system, device and medium for accelerating local full-text retrieval of ES files. Background Art

[0002] With the development of technology, the hard disk capacity of PC computers is getting larger and larger, and more and more files can be stored on the computer. As a result, the following problems have emerged: Sometimes, users need to spend a long time to retrieve and find a file. Since the file retrieval function provided by the Windows system is too slow, people hope to achieve fast retrieval of all folders and files on the disk, so many corresponding retrieval software has emerged, including Elasticsearch software. The implementation principle of Elasticsearch software mainly includes the following steps: First, users submit data to the Elasticsearch database, and then the corresponding statements are tokenized by a tokenization controller, and the weights and tokenization results are stored in the data together. When users search for data, the results are ranked and scored according to the weights, and then the returned results are presented to users. However, the process of submitting data to the Elasticsearch database is very time-consuming, which greatly affects the user experience. Summary of the Invention

[0003] To solve at least one of the technical problems existing in the prior art to a certain extent, an object of the present invention is to provide a method, system, device and medium for accelerating local full-text retrieval of ES files.

[0004] The technical solution adopted by the present invention is as follows:

[0005] A method for accelerating local full-text retrieval of ES files includes the following steps:

[0006] Collect the file information of folders on the disk by using the file record reading function of the fsutil tool;

[0007] Save the file information to the preset program memory and name it original;

[0008] Obtain the full path of the file on the disk according to original, process the full path, and obtain a file list;

[0009] Upload the file content on the disk to the Elasticsearch database according to the file list;

[0010] Input the keyword to be retrieved, and obtain the retrieval result according to the keyword and the Elasticsearch database.

[0011] Further, the file information includes file name, file ID, parent file ID, and timestamp;

[0012] Saving the file information to a preset program memory and naming it original includes:

[0013] Using a HashMap object, taking the file ID as the key and the file name as the value, saving it to a preset program memory, and naming it original.

[0014] Further, obtaining the full path of the file on the disk according to original, processing the full path to obtain a file list, including:

[0015] Putting the full path into a TreeMap object and sorting the TreeMap object;

[0016] According to the sorted TreeMap object, storing the full paths of all files into a HashMap object and naming it fileListMap;

[0017] Taking out all file lists from fileListMap according to the folders on the disk and naming it fileList.

[0018] Further, there is a mapping relationship (key, value) between the full path stored in the HashMap object and the file. Among them, the key represents the full path of the folder, and the value represents the List object in which the files saved with the full path as the prefix are stored.

[0019] Further, uploading the file content on the disk to the Elasticsearch database according to the file list includes:

[0020] Filtering the suffix of each file according to the file list to obtain the text content included in the files in the preset format;

[0021] Using the createIndex method to insert the obtained text content into the Elasticsearch database.

[0022] Further, the preset format includes Word format, PDF format, TXT format, Excel format, PPT format, and HTML format;

[0023] Obtaining the text content included in the files in the preset format includes:

[0024] Using a multi-threaded method to parallelly obtain the text content included in the files in the preset format.

[0025] Further, the input keyword to be retrieved, and the retrieval result is obtained according to the keyword and the Elasticsearch database, including:

[0026] Input the keyword to be retrieved;

[0027] According to the keyword, use the QueryBuilders.matchPhraseQuery method to perform index retrieval in the Elasticsearch database to obtain the retrieval result.

[0028] Another technical solution adopted by the present invention is:

[0029] A system for accelerating local full-text retrieval of ES files, including:

[0030] An information collection module, which is used to collect the file information of the folders on the disk by using the file record reading function of the fsutil tool;

[0031] An information storage module, which is used to save the file information to the preset program memory and name it original;

[0032] A path acquisition module, which is used to obtain the full path of the file on the disk according to original, process the full path, and obtain a file list;

[0033] A path upload module, which is used to upload the file content on the disk to the Elasticsearch database according to the file list;

[0034] A retrieval module, which is used to input the keyword to be retrieved, and obtain the retrieval result according to the keyword and the Elasticsearch database.

[0035] Another technical solution adopted by the present invention is:

[0036] A device for accelerating local full-text retrieval of ES files, including:

[0037] At least one processor;

[0038] At least one memory, which is used to store at least one program;

[0039] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0040] Another technical solution adopted by the present invention is:

[0041] A storage medium, in which a program executable by a processor is stored, and the program executable by the processor is used to execute the above method when executed by the processor.

[0042] The beneficial effects of the present invention are as follows: The present invention uses fsutil to view USN-related information and initializes it into a memory file path mapping Map (storing the directory as the key and the list of all files under the directory as the value). All local file paths are found and their contents are extracted and stored in the Elasticsearch database, enabling direct retrieval during full-text retrieval using the relevant Elasticsearch APIs; this greatly improves the efficiency of data upload. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following introduces the accompanying drawings related to the technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings below are only for conveniently and clearly presenting some embodiments of the technical solutions in the present invention. For those skilled in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0044] Figure 1 is a flowchart of the steps of a method for accelerating local full-text retrieval of ES files in an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of file information in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] The following details the embodiments of the present invention. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as a limitation of the present invention. For the step numbers in the following embodiments, they are only set for the convenience of explanation and illustration, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0047] In the description of the present invention, it should be understood that for the orientation description, such as the orientation or positional relationship indicated by up, down, front, back, left, right, etc., is based on the orientation or positional relationship shown in the accompanying drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0048] In the description of the present invention, "several" means one or more, "multiple" means more than two, and understandings such as "greater than", "less than", "exceeding", etc. do not include the corresponding number, while understandings such as "above", "below", "within", etc. include the corresponding number. If there is a description of "first" and "second", it is only for the purpose of distinguishing technical features and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features.

[0049] In the description of the present invention, unless otherwise clearly defined, words such as "set", "install", "connect", etc. should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0050] As Figure 1 described, this embodiment provides a method for accelerating the local full-text retrieval of ES files, including the following steps:

[0051] S1. Use the file record reading function of the fsutil tool to collect the file information of the folders on the disk.

[0052] According to the disk where the folder is located, use the file record reading function of the fsutil system tool to collect file records. For example, use the command "fsutil usn readJournal E:" to collect all file records on the E disk, and the collected information is as Figure 2 shown.

[0053] S2. Save the file information to the preset program memory and name it original.

[0054] Referring to Figure 2, the collected information includes key information such as file name, file ID, parent file ID, timestamp, etc. We use the HashMap object to store the file ID as the key and the file name as the value, and save it to the program memory named original for convenient processing in the next step.

[0055] S3. Obtain the full path of the file on the disk according to original, process the full path, and obtain a file list.

[0056] Read the original, loop through each file, and use the relationship between the file ID and the parent file ID to splice and restore the full path (absolute path) of the file on the disk, and store the path in a TreeMap object. Sort the TreeMap object and process it into a HashMap object named fileListMap, where the key is the full path (absolute path) of the folder and the value is a List object containing all the files with this path as a prefix. Retrieve all the file lists from fileListMap according to the passed-in folder path and name it fileList.

[0057] S4. Upload the file content on the disk to the Elasticsearch database according to the file list.

[0058] Among them, step S4 is implemented in a multi-threaded manner, including steps S41 - S42:

[0059] S41. Filter the suffix of each file according to the file list to obtain the text content contained in the preset format files.

[0060] Read fileList; filter the suffix of each file and extract the text content contained in files such as Word, PDF, TXT, Excel, PPT, and HTML.

[0061] S42. Use the createIndex method to insert the obtained text content into the Elasticsearch database.

[0062] Insert the extracted text content into the Elasticsearch database using the createIndex (encapsulated method for inserting into the Elasticsearch database) method.

[0063] S5. Input the keyword to be retrieved and obtain the retrieval result according to the keyword and the Elasticsearch database.

[0064] Among them, step S5 includes steps S51 - S53:

[0065] S51. Input the keyword to be retrieved;

[0066] S52. Use the QueryBuilders.matchPhraseQuery() method to retrieve keywords in the index library;

[0067] S53. Obtain the retrieval result and display it.

[0068] In summary, this embodiment can greatly improve the retrieval speed of implementing full-text retrieval of local disks using Elasticsearch. In the early stage, the fsutil tool is used to view USN-related information and initialize it as a memory file path mapping Map (storing the directory as the key and the list of all files in the directory as the value). All local file paths are found and their contents are extracted and stored in the Elasticsearch database. When performing full-text retrieval using the Elasticsearch-related API, direct retrieval can be carried out. The NTFS file records initialize the file directory and find all file lists, greatly improving the speed of retrieving directory files compared to traditional recursive browsing of multi-level directories, thereby achieving an improvement in the speed of implementing full-text retrieval of local disks using Elasticsearch.

[0069] This embodiment also provides a system for accelerating local full-text retrieval of ES files, including:

[0070] An information collection module for collecting file information of folders on the disk using the file record reading function of the fsutil tool;

[0071] An information storage module for saving the file information to the preset program memory and naming it original;

[0072] A path acquisition module for obtaining the full path of the files on the disk according to original, processing the full path, and obtaining a file list;

[0073] A path upload module for uploading the file contents on the disk to the Elasticsearch database according to the file list;

[0074] A retrieval module for inputting the keywords to be retrieved and obtaining retrieval results according to the keywords and the Elasticsearch database.

[0075] The system for accelerating local full-text retrieval of ES files in this embodiment can execute a method for accelerating local full-text retrieval of ES files provided by the method embodiment of the present invention, can execute any combination of the implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0076] This embodiment also provides a device for accelerating local full-text retrieval of ES files, including:

[0077] At least one processor;

[0078] At least one memory for storing at least one program;

[0079] When the at least one program is executed by the at least one processor, the at least one processor is caused to implement asFigure 1 The method shown

[0080] An apparatus for accelerating local full-text retrieval of ES files in this embodiment can execute a method for accelerating local full-text retrieval of ES files provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0081] This application embodiment also discloses a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 The method shown

[0082] This embodiment also provides a storage medium, which stores instructions or programs that can execute a method for accelerating local full-text retrieval of ES files provided by the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method are provided.

[0083] In some alternative embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two consecutive blocks shown can actually be executed substantially simultaneously or the blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated, in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0084] In addition, although the present invention has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It should also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. Rather, given the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Thus, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It should also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0085] If the described function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0086] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in connection with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0087] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable media can even be paper or other suitable media on which the program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then storing it in a computer memory.

[0088] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), and the like.

[0089] In the above description of this specification, the description with reference to terms such as "one embodiment / Example", "another embodiment / Example", or "certain embodiments / Examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0090] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and purposes of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

[0091] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for accelerating local full-text retrieval of ES files, characterized in that, It includes the following steps: Use the file record reading function of the fsutil tool to view the USN and collect the file information of the folders on the disk. The file information includes file name, file ID, parent file ID, and timestamp; Save the file information to the preset program memory and name it original. Among them, saving the file information to the preset program memory and naming it original includes: using the HashMap object to use the file ID as the key and the file name as the value, and save it to the preset program memory and name it original; Obtain the full path of the file on the disk according to original, process the full path to obtain a file list. Among them, obtaining the full path of the file on the disk according to original and processing the full path to obtain a file list includes: reading original, and loop processing each file to splice and restore the full path of the file on the disk using the relationship between the file ID and the parent file ID; store the full path in the TreeMap object, sort the TreeMap object; store the full paths of all files in the HashMap object according to the sorted TreeMap object and name it fileListMap; take out all file lists from fileListMap according to the folders on the disk and name it fileList; Upload the file content on the disk to the Elasticsearch database according to the file list; Input the keyword to be retrieved, and obtain the retrieval result according to the keyword and the Elasticsearch database.

2. The method for accelerating local full-text retrieval of ES files according to claim 1, wherein, There is a mapping relationship (key, value) between the full path stored in the HashMap object and the file, where key represents the full path of the folder and value represents the List object in which the files with the full path as the prefix are saved.

3. A method for accelerating local full-text retrieval of ES files according to claim 1, characterized in that The uploading the file content on the disk to the Elasticsearch database according to the file list includes: Perform suffix filtering on each file according to the file list to obtain the text content contained in the files in the preset format; Use the createIndex method to insert the obtained text content into the Elasticsearch database.

4. A method for accelerating local full-text retrieval of ES files according to claim 3, characterized in that The preset format includes Word format, PDF format, TXT format, Excel format, PPT format, and HTML format; The obtaining the text content contained in the files in the preset format includes: Use the method of multi-threading to parallelly obtain the text content contained in the files in the preset format.

5. A method for accelerating local full-text retrieval of ES files according to claim 1, characterized in that, The inputting the keyword to be retrieved and obtaining the retrieval result according to the keyword and the Elasticsearch database includes: Input the keyword to be retrieved; According to the keyword, use the QueryBuilders.matchPhraseQuery method to perform index retrieval in the Elasticsearch database to obtain the retrieval result.

6. A system for accelerating local full-text retrieval of ES files, characterized in that, It includes: An information collection module, which is used to view the USN by using the file record reading function of the fsutil tool and collect the file information of the folders on the disk. Among them, the file information includes file name, file ID, parent file ID, and timestamp; An information storage module, which is used to save the file information to the preset program memory and name it original. Among them, saving the file information to the preset program memory and naming it original includes: using a HashMap object to use the file ID as the key and the file name as the value, saving it to the preset program memory and naming it original; A path acquisition module, which is used to obtain the full path of the file on the disk according to original, process the full path, and obtain a file list. Among them, obtaining the full path of the file on the disk according to original and processing the full path to obtain a file list includes: reading original, looping through each file and splicing and restoring the full path of the file on the disk by using the relationship between the file ID and the parent file ID; storing the full path in a TreeMap object, sorting the TreeMap object; storing the full paths of all files in a HashMap object according to the sorted TreeMap object and naming it fileListMap; taking out all file lists from fileListMap according to the folders on the disk and naming it fileList; A path upload module, which is used to upload the file content on the disk to the Elasticsearch database according to the file list; A retrieval module, which is used to input the keyword to be retrieved and obtain the retrieval result according to the keyword and the Elasticsearch database.

7. A device for accelerating local full-text retrieval of ES files, characterized in that, Including: At least one processor; At least one memory, which is used to store at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1-5.

8. A storage medium storing a program executable by a processor, characterized in that, The program executable by the processor is used to execute the method according to any one of claims 1-5 when executed by the processor.

Images