File retrieval method, device, equipment and storage medium

Abstract

The present invention discloses a file retrieval method, apparatus, device and storage medium, and relates to the field of data retrieval technology, wherein the method comprises: the user state space responds to the file retrieval instruction received at the current moment, determines the target retrieval strategy corresponding to the file retrieval instruction, and updates the event filtering strategy to the kernel state space; the kernel state space obtains the target file operation event from all file operation events captured at the current moment based on the updated event filtering strategy; the user state space determines the metadata information of the target file corresponding to the target file operation event, and updates the hybrid index database based on the metadata information, and then searches in the updated hybrid index database based on the target retrieval strategy and the file retrieval instruction to obtain the file retrieval result. The present invention realizes the real-time update of file system metadata information and the efficient retrieval of multiple retrieval dimensions, meets the efficient file retrieval requirements, and improves the file retrieval efficiency.

Description

Technical Field

[0001] The present disclosure generally relates to the field of data retrieval technology, and more particularly to a file retrieval method, apparatus, device, and storage medium. Background Art

[0002] As we all know, file retrieval technology is a core basic function in the field of operating systems and data management. However, the current mainstream file retrieval method is usually based on retrieval tools, such as find tools, mlocate tools, and mlocate tools. Due to the problems of these retrieval tools relying on specific file systems, relying on periodic full disk scanning or real-time traversal, and having high retrieval overhead, the existing index library built based on real-time traversal and periodic scanning of the file system has certain limitations for file retrieval. Summary of the Invention

[0003] In view of the above-mentioned defects or deficiencies in the prior art, it is desired to provide a file retrieval method, apparatus, device and storage medium, which provide an efficient real-time file retrieval tool through a collaborative architecture based on a zero-copy data capture mechanism driven by kernel-mode file operation events and a dynamic hybrid index in user-mode, thereby achieving real-time update of file system metadata information and efficient retrieval in multiple retrieval dimensions, thereby not only meeting the needs of efficient file retrieval, but also improving file retrieval efficiency.

[0004] In a first aspect, the present invention provides a file retrieval method, which is applied to a computer device, wherein the computer device is installed with an operating system, wherein the operating system includes a user state space and a kernel state space; the method comprises:

[0005] The user state space determines a target search strategy corresponding to the file search instruction in response to the file search instruction received at the current moment, and updates the event filtering strategy to the kernel state space;

[0006] The kernel state space obtains a target file operation event from all file operation events captured at the current moment based on the updated event filtering strategy;

[0007] The user space determines metadata information of the target file corresponding to the target file operation event, and updates the hybrid index database based on the metadata information; the updated hybrid index database includes a mapping relationship between the target file and the metadata information, and a mapping relationship between historical files and historical metadata information; and the hybrid index database supports multiple types of search conditions;

[0008] The user state space searches the updated hybrid index database based on the target search strategy and the file search instruction to obtain a file search result.

[0009] In conjunction with the first aspect, in a possible implementation, obtaining a target file operation event from all file operation events captured at the current moment based on the updated event filtering policy includes:

[0010] Based on the different levels of event filtering policies included in the updated event filtering policy, performing different levels of event filtering on all the file operation events to filter out the target file operation event from all the file operation events;

[0011] The event filtering strategies of different levels include a first-level event filtering strategy based on filtering based on a preset path prefix, and a second-level event filtering strategy based on filtering based on a preset file type and a preset file suffix.

[0012] In conjunction with the first aspect, in a possible implementation, updating the hybrid index database based on the metadata information includes:

[0013] In the case where the hybrid index database includes an inverted index library, a prefix tree index library and a metadata index library,

[0014] Based on the file ID and file name keywords of the target file, updating the existing keyword-file ID mapping relationship table in the inverted index library;

[0015] Based on the file ID of the target file and the file path in the metadata information, updating the existing file path-file ID mapping relationship table in the prefix tree index library;

[0016] Based on the metadata information and the file ID, the existing metadata information-file ID mapping relationship table in the metadata index library is updated.

[0017] In conjunction with the first aspect, in one possible implementation, performing a search in the updated hybrid index database based on the target search strategy and the file search instruction to obtain a file search result includes:

[0018] Based on the search condition corresponding to the file search instruction, determining at least two target index libraries from the updated inverted index library, the updated prefix tree index library, and the updated metadata index library;

[0019] Based on the target retrieval strategy, a collaborative search is performed in the at least two target index libraries to obtain the document retrieval result.

[0020] In conjunction with the first aspect, in a possible implementation, updating the event filtering policy to the kernel state space includes:

[0021] Obtaining the preset format rule configuration file at the current moment;

[0022] Parsing the event filtering rules from the preset format rule configuration file and performing a validity check on the event filtering rules;

[0023] The event filtering rules that pass the legality check are used as updated event filtering strategies and updated to the kernel state space in real time.

[0024] In conjunction with the first aspect, in a possible implementation, the method further includes:

[0025] In the case where the file retrieval instruction is a natural language retrieval instruction, processing the natural language retrieval instruction into a structured language retrieval instruction based on a pre-trained target language processing model;

[0026] Among them, the target processing model is obtained by training the NLP model based on the Transformer framework based on sample natural language retrieval instructions in different file retrieval scenarios and sample structured retrieval instructions identified by each sample natural language retrieval instruction; and each of the sample natural language retrieval instructions has undergone data enhancement.

[0027] In conjunction with the first aspect, in one possible implementation, determining the target retrieval strategy corresponding to the file retrieval instruction includes:

[0028] Parsing the search conditions of the file search instruction and determining a target search plan based on all the parsed search conditions;

[0029] Based on a preset mapping relationship between the retrieval plan and the retrieval strategy, the target retrieval strategy corresponding to the target retrieval plan is determined.

[0030] In a second aspect, the present invention further provides a file retrieval device. The device comprises:

[0031] a policy determination unit configured to determine, in response to a file retrieval instruction currently received in the user state space, a target retrieval policy corresponding to the file retrieval instruction, and update the event filtering policy to the kernel state space;

[0032] An event acquisition unit configured to acquire, in a kernel space, a target file operation event from all file operation events captured at the current moment based on an updated event filtering policy;

[0033] An index library update unit configured to determine metadata information of a target file corresponding to the target file operation event in the user space and update a hybrid index database based on the metadata information; the updated hybrid index database includes a mapping relationship between the target file and the metadata information, and a mapping relationship between historical files and historical metadata information; and the hybrid index database supports multiple types of search conditions;

[0034] The file retrieval unit is configured to perform a search in the updated hybrid index database in the user space based on the target retrieval strategy and the file retrieval instruction to obtain a file retrieval result.

[0035] In a third aspect, the present invention further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the method according to the first aspect when executing the computer program.

[0036] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0037] Embodiments of the present invention provide a file retrieval method, apparatus, device, and storage medium. In the file retrieval method, when a user-mode space responds to a file retrieval instruction received at a current moment, the user-mode space first determines a target retrieval strategy corresponding to the file retrieval strategy and updates an event filtering strategy to the kernel-mode space. The kernel-mode space then obtains a target file operation event from all file operation events captured at the current moment based on the updated event filtering strategy. The user-mode space then further determines metadata information of a target file corresponding to the target file operation event and updates a hybrid index database based on the metadata information. The updated hybrid index database is then searched based on the target retrieval strategy and the file retrieval instruction to obtain a file retrieval result. Because the updated hybrid index database includes a mapping relationship between the target file and metadata information, as well as a mapping relationship between historical files and historical metadata information, and because the hybrid index database supports multiple types of retrieval conditions, an efficient, real-time file retrieval tool is provided through a collaborative architecture based on a zero-copy data capture mechanism driven by kernel-mode file operation events and a dynamic hybrid index in user mode. This achieves real-time updates of file system metadata information and efficient retrieval across multiple retrieval dimensions, thereby not only meeting efficient file retrieval requirements but also improving file retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 FIG1 is a flowchart of a file retrieval method according to an embodiment of the present invention;

[0040] Figure 2 FIG2 is a second flow chart of a file retrieval method according to an embodiment;

[0041] Figure 3 FIG3 is a flowchart of a file retrieval method according to an embodiment;

[0042] Figure 4 FIG4 is a fourth flow chart of a file retrieval method according to an embodiment;

[0043] Figure 5 FIG5 is a fifth flow chart of a file retrieval method according to an embodiment;

[0044] Figure 6 is a structural block diagram of a file retrieval device in one embodiment;

[0045] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the invention are shown in the accompanying drawings.

[0047] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments. In addition, the term "and / or" herein is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first" and "second" in the description and claims of the embodiments of the present invention are used to distinguish different objects, rather than to describe a specific order of objects.

[0048] As we all know, file retrieval technology is a core basic function in the field of operating systems and data management. However, the current mainstream file retrieval method is usually based on retrieval tools, such as the find tool, the mlocate tool, and other tools such as the mlocate tool. Due to other problems such as the dependence of the above-mentioned retrieval tools on specific file systems, dependence on periodic full disk scanning or real-time traversal, and high retrieval overhead, the existing index library built based on real-time traversal and periodic scanning of the file system has certain limitations for file retrieval.

[0049] For example, the find tool is based on real-time traversal of the file system and checks file attributes one by one, which is suitable for one-time retrieval, but has poor retrieval performance.

[0050] The mlocate tool periodically scans the file system and builds an index library. Index library updates are delayed by several hours. Although searching the index directly during retrieval speeds up retrieval, its limitation is that the index library update cycle is fixed and cannot reflect file system changes in real time.

[0051] The FSearch tool is a graphical fast file search tool based on GTK3. Although it can achieve the purpose of fast search by pre-establishing a file index library, it cannot capture the dynamic changes of the file system, and index updates still need to be actively triggered. Among them, GTK is an open source, multi-platform graphical user interface (GUI) toolkit, and its full English name is GIMP Toolkit. The full Chinese and English names of GIMP are GNU Image Manipulation Program (GIMP). GUN is the abbreviation of GNU's Not Unix, which indicates that GNU is a Unix-like system, but not a Unix operating system. Toolkit is a toolkit or toolbox.

[0052] From this, we can understand that the above-mentioned traditional search tools have at least three technical bottlenecks:

[0053] The first technical bottleneck is the real-time performance flaw of traditional search tools: they rely on periodic full disk scans or real-time traversals, which leads to delayed index updates and makes it difficult to adapt to the dynamic changes in the file system environment.

[0054] The second technical bottleneck is resource efficiency: full disk scanning consumes a large amount of input / output (I / O) and central processing unit (CPU) resources, and retrieving metadata information based on stat system calls further increases overhead.

[0055] The third technical bottleneck is the single retrieval mode: it only supports retrieval based on structured conditions such as file name and file size, lacks natural language interaction capabilities, and cannot parse file retrieval instructions such as semantic requests such as "find Word files modified yesterday".

[0056] In today's digital age, businesses generate and process enormous amounts of data. Traditional management methods are no longer able to meet the needs of efficient file retrieval. Related technologies suffer from limitations such as low retrieval efficiency, delayed index updates, and a single retrieval dimension. Consequently, a file retrieval system is needed that makes information search and retrieval faster and more convenient, significantly improving work efficiency.

[0057] In order to solve the above technical problems, the present invention provides a file retrieval method, apparatus, equipment and storage medium, which provides an efficient real-time file retrieval tool through a zero-copy data capture mechanism driven by kernel-mode file operation events and a collaborative architecture of user-mode dynamic hybrid indexing, thereby achieving real-time update of file system metadata information and efficient retrieval in multiple retrieval dimensions. Therefore, it can not only meet the needs of efficient file retrieval, but also improve file retrieval efficiency.

[0058] The following combination Figures 1 to 7 The present invention describes a file retrieval method, apparatus, device, and storage medium. The file retrieval method can be applied to a computer device equipped with an operating system comprising a user-mode space and a kernel-mode space. The computer device can be a personal computer, a server, an embedded system, or other device. The present invention does not specifically limit this. Furthermore, the file retrieval method can also be applied to a file retrieval device provided in the computer device. The file retrieval device can be implemented using software, hardware, or a combination of both. The following describes the file retrieval method using a computer device as an example.

[0059] To facilitate understanding of the file retrieval method provided by the embodiments of the present invention, the file retrieval method provided by the present invention will be described in detail below through the following exemplary embodiments. It is understood that the following exemplary embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0060] In one embodiment, a file retrieval method is provided, such as Figure 1 As shown, the method includes the following steps 101 to 104.

[0061] Step 101: The user state space determines a target search strategy corresponding to the file search instruction in response to the file search instruction currently received, and updates the event filtering strategy to the kernel state space.

[0062] The file retrieval instruction may be an instruction automatically generated after a user inputs a file retrieval statement, and the instruction is used to instruct a computer device to execute a corresponding file retrieval operation.

[0063] The document search statement may be input in other ways, such as voice input or input on a device, etc. The present invention does not impose any specific limitation on this.

[0064] For example, when the user space of the computer device provides a visual human-computer interaction interface for the user, the current file search instruction can be received by the user inputting a file search statement in the search input area of ​​the visual human-computer interaction interface.

[0065] It is understandable that the user-mode space may include a retrieval engine module, and when the retrieval engine module responds to the file retrieval instruction received at the current moment, it first determines the target retrieval strategy corresponding to the file retrieval instruction. The strategy determination process here may include, but is not limited to, determining the database to be retrieved, the keywords to be retrieved, and clarifying the logical relationship and search steps between the search terms based on the analysis of the file retrieval instruction; thereby obtaining the target retrieval strategy corresponding to the file retrieval instruction. Alternatively, a corresponding file retrieval plan can be formulated based on the different metadata fields in the file retrieval instruction, and then the target retrieval strategy for implementing each file retrieval plan can be further determined. The present invention does not specifically limit the specific process of determining the target retrieval strategy.

[0066] In addition, the user-state space can also update the event filtering policy in response to the event filtering policy received at the current moment, and update the updated event filtering policy to the kernel-state space in real time; the update process here can use the Berkeley Packet Filter (BPF) hash table shared by the kernel-state space to dynamically update the event filtering policy to the kernel-state space, so that the updated event filtering policy can take effect immediately in the kernel-state space.

[0067] It should be noted that the BPF hash table can be used to achieve the purpose of hot updating of event filtering strategies, so as to facilitate the subsequent rapid and targeted screening of useful target file operation events.

[0068] Step 102: The kernel state space obtains a target file operation event from all file operation events captured at the current moment based on the updated event filtering policy.

[0069] Each file operation event may specifically be an event of performing a specified operation on an initial file. The kernel state space may specifically be a Linux kernel state space.

[0070] Specifically, for the kernel-state space, it can capture all initial operation events in real time and send them to the user-state space to update the mixed index database. It can also filter out target file operation events from all initial operation events captured at the corresponding moment based on the updated event filtering policy sent by the user-state space at a certain moment, and then send them to the user-state space to update the mixed index database. Therefore, when the kernel-state space receives the updated event filtering policy sent by the user-state space, it will filter all initial operation events captured at the current moment. Specifically, the event capture module built into the kernel-state space can first capture or collect file operation events; the event capture module here is responsible for capturing file operation events in real time, and transmitting the filtered target file operation events to the user-state space through the BPF ring buffer.

[0071] Furthermore, for all captured or collected file operation events, the event capture module can further use the updated event filtering strategy to filter irrelevant events, so as to obtain target file operation events that meet actual needs, and then push all target file operation events to the user space through the BPF ring buffer.

[0072] It's important to note that the kernel-mode system call layer and Virtual File System (VFS) layer inject an Extended Berkeley Packet Filter (eBPF) program, allowing for dynamic hooking of pre-configured kernel key functions. This allows the kernel-mode event capture module to capture all file operation events matching the hooked kernel key functions in real time, in response to event capture instructions. These events include file creation, file deletion, file renaming, file writing, file permission modification, file ownership modification, soft link creation, and hard link creation, among other file operation events.

[0073] For example, the kernel key functions related to file operation events are as follows:

[0074] vfs_create: monitors file creation events;

[0075] vfs_unlink: monitors file deletion events;

[0076] vfs_rename: monitors file name modification events;

[0077] vfs_write: monitors file write events;

[0078] sys_chmod: monitors file permission modification events;

[0079] sys_chown: monitors file owner modification events;

[0080] vfs_link: monitors file hard link creation events;

[0081] vfs_symlink: Monitors file soft link creation events.

[0082] Furthermore, an example of capturing a file operation event of a hooked kernel key function is as follows:

[0083] vfs_create: Monitors file creation events. After a file is successfully created, the file operation type event, file absolute path, file type, inode number, file size, number of disk blocks occupied by the file, file owner, file group, file permissions, last access time, last modification time, last status change time, file creation time, number of hard links, identifier of the device where the file is located, and other metadata information are encapsulated into a unified standardized event structure and sent to the event filtering unit in the event capture module.

[0084] vfs_unlink: Monitors file deletion events. After a file is successfully deleted, it encapsulates the file operation type event, file absolute path, file type and other metadata information into an event structure and sends it to the event filtering unit in the event capture module.

[0085] vfs_rename: Monitors file name modification events. After a file name modification is successful, it encapsulates metadata information such as the file operation type event, file absolute path, file type, original file absolute path, and last status change time into an event structure and sends it to the event filtering unit in the event capture module.

[0086] vfs_write: monitors file write events, intercepts system call exits, and after the file data is successfully written, encapsulates metadata information such as the file operation type event, file absolute path, file type, file size, last modification time, and last status change time into an event structure and sends it to the event filtering unit in the event capture module.

[0087] sys_chmod: Monitors file permission modification events. After a file permission modification is successful, it encapsulates metadata information such as the file operation type event, file absolute path, file type, file permissions, and the last status change time into an event structure and sends it to the event filtering unit in the event capture module.

[0088] sys_chown: Monitors file owner modification events. After a file owner modification is successful, it encapsulates metadata information such as the file operation type event, file absolute path, file type, file owner, file group, and last status change time into an event structure and sends it to the event filtering unit in the event capture module.

[0089] vfs_link: Monitors file hard link creation events. After a file hard link is successfully created, the metadata information of the encapsulated event is the information of the newly created hard link file. The event structure encapsulation content is the same as the file creation event and is sent to the event filtering unit in the event capture module.

[0090] vfs_symlink: Monitors file soft link creation events. After a file soft link is successfully created, the metadata information of the encapsulated event is the information of the newly created soft link file. The event structure encapsulation content is the same as the file creation event. At the same time, the absolute path information of the soft link's target file is added and sent to the event filtering unit in the event capture module.

[0091] In this embodiment, examples of relevant fields of the standardized event structure are:

[0092] event_type: event type;

[0093] absolute_path: absolute path of the file;

[0094] src_absolute_path: the absolute path of the file before modification;

[0095] dst_absolute_path: absolute path of the modified file;

[0096] softlink_absolute_path: the absolute path of the target file of the soft link;

[0097] type: file type;

[0098] inode: inode number;

[0099] size: file size;

[0100] blocks: the number of disk blocks occupied by the file;

[0101] uid: the owner of the file;

[0102] gid: the group to which the file belongs;

[0103] mode: file permissions;

[0104] access: the last access time;

[0105] modify: the last modification time;

[0106] change: the time of the last status change;

[0107] birth: file creation time;

[0108] links: the number of hard links;

[0109] device: The identifier of the device where the file is located.

[0110] In this way, the event filtering unit in the event capture module uses the BPF hash table to receive the updated event filtering strategy and filters irrelevant events for all captured or collected file operation events accordingly, and then sends all target file operation events obtained after filtering to the event propagation unit in the event capture module. The event propagation unit pushes all target file operation events to the user space through the BFP ring buffer.

[0111] The event propagation unit is responsible for transmitting valid target file operation events to user space. File operation events occur continuously in the operating system. By accumulating multiple file operation events in kernel space and submitting them in batches, the frequency of switching between kernel and user space is reduced. Correctly sequencing file operation events monitored in kernel space is crucial. Using the BPF ring buffer, based on a shared buffer, we ensure the sequential nature of all acquired target file operation events, ensuring that all target file operation events are processed by user space in the order in which they occur, while also achieving zero-copy data synchronization.

[0112] It should be noted that the event capture module is a core component of the operating system. It is responsible for capturing file operation events in real time in the kernel space, and passing all filtered target file operation events to the index management module in the user space through efficient event filtering strategies, supporting dynamic configuration of event filtering strategies.

[0113] Step 103: The user-mode space determines the metadata information of the target file corresponding to the target file operation event, and updates the hybrid index database based on the metadata information; the updated hybrid index database includes the mapping relationship between the target file and the metadata information, as well as the mapping relationship between the historical file and the historical metadata information; and the hybrid index database supports multiple types of retrieval conditions.

[0114] It should be noted that the user-mode space can also include an index management module, which receives target file operation events from the event capture module to the application layer, and updates the file metadata information to the index management module in real time; in addition, the index management module, as the core data processing unit of the operating system, is responsible for structured organization and efficient retrieval of file data; and, the index management module is based on a hybrid index structure to realize real-time and rapid associated retrieval of file metadata.

[0115] For the index management module, it first receives all target file operation events sent from the kernel space from the BPF ring buffer, and then classifies and sorts all target file operation events through the parallel update control mechanism of the shard queue architecture, so as to ensure that a category of target file operation events corresponding to the same file are strictly processed in the order of occurrence, and target file operation events corresponding to different classified files are processed in parallel, realizing parallel updates of file-level serial, operation-level merging and shard-level parallelism; thereby obtaining the processed target files; the number of target files is the same as the number of categories of target file operation events and corresponds one to one.

[0116] Thereafter, the hybrid index database is updated based on the metadata information of each target file; thereby, the updated hybrid index database includes not only the mapping relationship between each target file and its metadata information, but also the mapping relationship between all historical files and their respective historical metadata information.

[0117] It should be noted that, when a mixed index database that supports conditional retrieval of multiple data types is pre-built in the index management module, the mixed index database can be updated based on the metadata information of each target file; conversely, when the mixed index database is not retrieved in the index management module, the full index database of the file system can be constructed first, and then the mixed index structure of the full index database can be involved to build a mixed index database that can support conditional retrieval of multiple data types, and then the mixed index database can be updated based on the metadata information of each target file.

[0118] The hybrid index database can support conditional retrieval based on multiple data types such as file name, file path, file size, modification time and full metadata information, that is, it supports multi-dimensional efficient retrieval.

[0119] Step 104: The user space searches the updated hybrid index database based on the target search strategy and the file search instruction to obtain a file search result.

[0120] Specifically, the search engine module in the user space can receive the file search instruction input by the user and determine the target search strategy corresponding to the file search instruction, and complete the data search through the updated hybrid index database in the index management module to obtain the file search result. That is, the file search instruction is parsed into multiple search tasks that support parallelization.

[0121] Exemplarily, the retrieval engine module may include a unified interface access unit, a structured condition processing unit and a parallel retrieval execution unit. The unified interface access unit provides multiple types of standardized interfaces. The structured condition processing unit parses the file retrieval instructions into multiple retrieval tasks that support parallelization. The parallel retrieval execution unit schedules multiple back-end data sources in parallel to collaboratively retrieve file metadata and merge multi-source data, that is, through task decomposition, asynchronous pipeline, and lock-free communication, multiple index data sources in the hybrid index database that have been collaboratively updated in multiple retrieval plans achieve efficient parallel collaboration, thereby obtaining file retrieval results.

[0122] The unified interface access unit supports multiple client access methods such as terminal command, desktop, and Web, and supports interfaces such as local Software Development Kit (SDK), Remote Procedure Call (RPC), and Representational State Transfer Application Program Interface (RESTful API), thereby providing a unified interface access layer with good compatibility and ease of use, and improving retrieval efficiency through parallel retrieval optimization.

[0123] It should be noted that the user state space may also include a visual interaction module, which is displayed in a visual human-computer interaction interface, not only providing good human-computer interaction and intuitive display of file retrieval results, but also supporting multi-mode hybrid retrieval.

[0124] The visualization interaction module has visualization interaction function, and the visualization interaction function completes the visualization interaction function by communicating and cooperating with the search engine module, providing a good human-computer interaction environment.

[0125] As for the visual interaction module, it has multi-modal retrieval function and intuitive result display function.

[0126] The multi-mode search function supports natural language search, find command format search, structured conditional language search and mixed mode search.

[0127] The intuitive result display function supports structured presentation of file metadata and is deeply integrated with the system file browser.

[0128] For example, in the visual human-computer interaction interface, there are three areas: search input area, function button area and search result display area. The search input area dynamically recommends high-frequency search terms based on historical search logs and file system metadata, and the search input area can switch between different search modes; the function button area provides search buttons and mode switching buttons; the search result display area can support keyword matching highlighting for file search results, support expanding file metadata information, and realize "zero-click" path jump through deep system integration (double-clicking the path column automatically calls up the operating system's default file manager to locate the file entity).

[0129] The file retrieval method provided by an embodiment of the present invention is as follows: when the user-state space responds to a file retrieval instruction received at the current moment, it first determines the target retrieval strategy corresponding to the file retrieval instruction and updates the event filtering strategy to the kernel-state space. The kernel-state space obtains the target file operation event from all file operation events captured at the current moment based on the updated event filtering strategy; the user-state space further determines the metadata information of the target file corresponding to the target file operation event and updates the hybrid index database based on the metadata information. Then, based on the target retrieval strategy and the file retrieval instruction, the updated hybrid index database is searched to obtain the file retrieval result. Since the updated hybrid index database includes a mapping relationship between the target file and metadata information, as well as a mapping relationship between historical files and historical metadata information; and the hybrid index database supports multiple types of retrieval conditions, an efficient real-time file retrieval tool is provided by a collaborative architecture based on a zero-copy data capture mechanism driven by kernel-state file operation events and a dynamic hybrid index in user state, achieving the real-time update of file system metadata information and the efficient retrieval purpose of multiple retrieval dimensions. Thus, it can not only meet the needs of efficient file retrieval but also improve file retrieval efficiency.

[0130] Based on the above Figure 1 In the method shown in FIG. 1 , in an exemplary embodiment, in step 101, the event filtering policy is updated to the kernel state space. The specific process in this embodiment can be achieved by Figure 2 Steps 201 to 203 shown are implemented as follows.

[0131] Step 201: Obtain the current preset format rule configuration file.

[0132] Step 202: parse the event filtering rules from the preset format rule configuration file, and perform a validity check on the event filtering rules.

[0133] Step 203: The event filtering rules that have passed the validity check are used as updated event filtering policies and updated to the kernel state space in real time.

[0134] It should be noted that in addition to the index management module and the retrieval engine module, the user-state space can also include a dynamic rule management module. The dynamic rule management module supports the user-state space to modify the filtering configuration file, and through the use of the BPF hash table opened by the event capture module in the kernel-state space for data interaction, the event filtering policy is dynamically updated in real time to the event capture module in the kernel-state space, so that the filtered event filtering policy takes effect immediately.

[0135] Specifically, the dynamic rule management module loads the event filtering strategy into the BPF hash table when the event capture module is initialized and runs, and can also dynamically update the event filtering strategy during the running of the event capture module.

[0136] In this way, the filter configuration file modified by the user in the user state space can be first obtained, and then the preset format rule configuration file at the current moment is read from the filter configuration file, such as reading the rule configuration file based on the YAML format.

[0137] For example, the content of the rule configuration file in this embodiment is as follows:

[0138] rule_set:

[0139] - {prefix: [" / home", " / var / log"], suffix: [".log", ".db"], type: [1,2]}.

[0140] For the preset format rule configuration file at the current moment, the event filtering rules can be parsed from the preset format rule configuration file first, and then the event filtering rules can be further verified for legitimacy. The event filtering rules that pass the legitimacy verification are used as the updated event filtering strategy and updated in real time to the BPF hash table in the kernel space in a hot update manner.

[0141] Based on the above Figure 1 In the method shown, in an exemplary embodiment, in step 102, the target file operation event is obtained from all file operation events captured at the current moment based on the updated event filtering policy. The specific determination process can be implemented through the following steps in this embodiment.

[0142] Based on the different levels of event filtering policies included in the updated event filtering policy, all file operation events are subjected to different levels of event filtering to filter out target file operation events from all file operation events.

[0143] Among them, different levels of event filtering strategies include a first-level event filtering strategy based on preset path prefix filtering, and a second-level event filtering strategy based on preset file types and preset file name suffixes.

[0144] Specifically, the kernel-mode space implements a two-level event filtering mechanism in the event capture module. This mechanism filters out irrelevant events based on preset path prefixes, file suffixes, and file types. The BPF hash table (BPF_MAP_TYPE_HASH) is used to enable the dynamic rule management module to dynamically update event filtering policies to the event capture module. Furthermore, the BPF ring buffer (BPF_MAP_TYPE_RINGBUF) is used to enable efficient data transfer between the event capture module and the index management module, achieving zero-copy data synchronization. The BPF ring buffer ensures event order by sending target file operation events to a shared buffer, making it particularly suitable for scenarios requiring high-frequency event collection and low-latency processing.

[0145] For example, the preset path prefix may be / home, the preset file name suffix may be .log, and the preset file type may be a regular file, a directory, and the like.

[0146] It should be noted that all file operation events received by the event filtering unit in the event capture module include the absolute path and file type of the operated file.

[0147] In this way, all captured file operation events are first filtered based on the file path, that is, filtering based on the preset path prefix, filtering by string prefix matching. Then, the matching file operation events are further filtered based on the preset file name suffix and preset file type. Both levels of filtering here are dynamic filtering processes, and irrelevant operation events are deleted through the two-level filtering mechanism.

[0148] Furthermore, file operation events that are successfully matched by both levels of filtering are directly discarded in the kernel state space, and file operation events that are not successfully matched by at least one level of filtering in the two levels of filtering are treated as target file operation events that need to be passed to the user state space, and are transmitted to the index management module of the user state space through the BPF ring buffer.

[0149] Based on the above Figure 1 In the method shown in FIG. 1 , in an exemplary embodiment, in step 103, the hybrid index database is updated based on the metadata information of the target file. The specific determination process is as follows: In this embodiment, when the hybrid index database includes an inverted index library, a prefix tree index library, and a metadata index library, the hybrid index database can be updated by Figure 3 Steps 301 to 303 are implemented as shown.

[0150] Step 301: Based on the file ID and file name keywords of the target file, the existing keyword-file ID mapping relationship table in the inverted index library is updated.

[0151] Step 302: Based on the file ID of the target file and the file path in the metadata information, the existing file path-file ID mapping relationship table in the prefix tree index library is updated.

[0152] Step 303: Based on the metadata information and the file ID, the metadata information-file ID mapping relationship table existing in the metadata index library is updated.

[0153] Specifically, the index management module of the user space is based on a hybrid index structure to construct a hybrid index database of an inverted index library, a prefix tree index library and a metadata index library, and the index management module can specifically include an index initialization unit, an event reception and concurrency control unit, an inverted index unit based on a file name, a prefix tree index unit based on a file path and a file metadata database storage unit.

[0154] The following is an exemplary description of the construction process of the hybrid index database in combination with the above units:

[0155] The index initialization unit is used to perform a full index construction for the existing file system of the current operating system when the operating system is run for the first time and the initialized database file has not been established. That is, the full index database of the file system is constructed for the first time. The construction process can refer to the existing construction method and is not specifically limited or explained here.

[0156] The event receiving and concurrency control unit is used to receive all target file operation events sent from the kernel space from the BPF ring buffer, and ensure that at least one target file operation event corresponding to the same initial file is strictly processed in the order of occurrence; the specific processing process can adopt a shard queue structure to ensure that a class of target file operation events corresponding to the same initial file are strictly processed in order, and different initial files are processed in parallel with the maximum degree of parallelism.

[0157] For shard queues, each shard corresponds to an independent waiting queue, and file operation events with the same file path are always routed to the same shard. Furthermore, concurrency control includes shard-level parallelism, file-level serialization, and operation-level merging. Shard-level parallelism allows different shard queues to be processed in parallel, file-level serialization ensures that file operation events for the same file are strictly executed sequentially, and operation-level merging merges multiple identical operations within a window period to reduce the number of index updates.

[0158] The inverted index unit based on the file name is used to segment the file name to obtain keywords, establish a hash table, map the keywords to a set of file IDs, realize the bidirectional mapping between the file name keywords and the files, and obtain an inverted index database containing a keyword-file ID mapping relationship table.

[0159] The file path-based prefix tree index unit is used to build a prefix tree index library based on the file path and file ID. It supports complex path wildcard retrieval and also supports fast retrieval from file path to file entity attributes. In the prefix tree index library, each node is a file, and each node is mapped to the file ID in the file metadata database, thereby obtaining a prefix tree index library containing a file path-file ID mapping relationship table.

[0160] The file metadata database storage unit is used to store the full file metadata information of all files and supports conditional retrieval based on the metadata information. That is, a complete information library is established based on the file metadata information of all files. Each file has a unique file ID in the information library. The information library establishes an index structure based on key metadata fields to support fast conditional filtering and retrieval of metadata information, such as file size, modification events, etc.; in this way, a full file metadata database is constructed, and the file metadata includes a full metadata information-file ID mapping relationship table.

[0161] It should be noted that, for the index management module, in addition to constructing the index layer of the hybrid index database, it can also include a cache layer that optimizes the response performance in high-frequency retrieval scenarios. The cache layer is used to cache file IDs whose retrieval frequency is greater than the preset frequency threshold within a preset time period, and sort the retrieval priority of each cached ID based on the cumulative retrieval frequency, thereby obtaining a file ID-cumulative retrieval frequency-retrieval priority mapping relationship table, and cache the corresponding file data if the storage capacity threshold is not exceeded; when the cache capacity exceeds the storage capacity threshold, the file ID with the lowest cumulative retrieval frequency within the preset time, or the file IDs and corresponding file data of the last few lowest cumulative retrieval frequencies can be deleted. In this way, each sub-unit in the index management module can significantly reduce resource consumption while ensuring real-time performance through layered collaboration and asynchronous update strategies.

[0162] Furthermore, a full file metadata database is constructed at the index layer based on the file metadata information. Each file is assigned a unique file ID in the file metadata database. In this way, when other indexing methods are used to obtain file retrieval results, the corresponding file metadata information can also be found according to the file metadata database.

[0163] In this embodiment, the content of the file metadata database can be described as follows:

[0164] There is a file named ssh-keygen. After word segmentation of the file name, two keywords, ssh and keygen, are obtained. Then, there are two index keys, ssh and keygen, in the inverted index database. At the same time, these two index keys may also be associated with other files. The index structure is shown as follows:

[0165] There are three files. The sample information stored in the file metadata database is as follows: Example 2:

[0166] [{"ID":1,"PATH":" / usr / bin / ssh-keygen",…},{"ID":2,"PATH":" / usr / sbin / ntp-keygen",…}, {"ID":3,"PATH":" / usr / bin / ssh-copy-id",…},…].

[0167] For the inverted index unit based on file name, after segmenting each file name, it creates an inverted index based on multiple segmentations and the file ID of the corresponding file to support fast search for files based on file name.

[0168] In this embodiment, the content of the inverted index unit is described in Example 3 below:

[0169] Based on the above example, the following inverted index structure is constructed for the files / usr / bin / ssh-keygen, / usr / bin / ssh-copy-id, and / usr / sbin / ntp-keygen:

[0170] "ssh": [1, 3, ...],

[0171] "keygen": [1,2, ...]

[0172] In addition, in the inverted index database, the keywords obtained by segmenting the file name can be used as keys, and the unique key in the file metadata database can be used as values ​​to establish a mapping.

[0173] Furthermore, for the prefix tree index unit based on the file path, the prefix tree index library it constructs contains a prefix tree structure of the complete path, and each node stores the file ID corresponding to the file in the path from the root node to the node; and the complete file path is stored in the prefix tree structure, supporting efficient prefix matching retrieval (such as / var / log / *); as in Example 1, after retrieving the associated file based on the keyword ssh, the file metadata information is retrieved in the prefix tree structure based on the corresponding file path; at the same time, it also supports file retrieval directly based on the prefix tree index library.

[0174] In this embodiment, the content of the prefix tree index library is described in Example 4 below:

[0175] Based on the above example, the following prefix tree structure is constructed for the files / usr / bin / ssh-keygen, / usr / bin / ssh-copy-id, and / usr / sbin / ntp-keygen:

[0176] / (root directory)("id": xx)

[0177] └── "bin" ("id": xx)

[0178] └── "ssh-keygen"("id": 1)

[0179] └── "ssh-copy-id" ("id": 3)

[0180] └── "sbin" ("id": xx)

[0181] └── "ntp-keygen" ("id": 2)

[0182] Optionally, in an embodiment of the present invention, the cache layer in the index management module is a performance acceleration layer of the index management module, which reduces the frequency of direct access to the underlying storage by caching file index data that is frequently accessed or retrieved, thereby optimizing the retrieval response speed.

[0183] At this time, the existing keyword-file ID mapping relationship table in the inverted index library can be updated according to the file ID and file name keyword of each target file; the existing file path-file ID mapping relationship table in the prefix tree index library can be updated according to the file wildcard and the file ID of each target file, and the file path in the metadata information of each target file; and the existing metadata information-file ID mapping relationship table in the metadata index library can be updated according to the metadata information and file ID of each target file; thereby obtaining an updated hybrid index database.

[0184] The embodiment of the present invention designs a hybrid index database in the user space, integrating the inverted index, prefix tree and metadata database. The hybrid index database supports keyword matching, wildcard search and multi-condition combination search, and combines the cache mechanism with parallel search optimization to achieve the purpose of rapid response to complex search conditions.

[0185] Based on the above Figure 1In the method shown in FIG. 1 , in an exemplary embodiment, in step 104, based on the target search strategy and the file search instruction, a search is performed in the updated hybrid index database to obtain a file search result. The specific search process in this embodiment can be performed by Figure 4 Steps 401 to 402 are implemented as shown.

[0186] Step 401: Based on the search condition corresponding to the file search instruction, determine at least two target index repositories from the updated inverted index repositories, the updated prefix tree index repositories, and the updated metadata index repositories.

[0187] Step 402: Based on the target retrieval strategy, a collaborative search is performed in at least two target index repositories to obtain a document retrieval result.

[0188] Specifically, for the search engine module in the user space, it first determines whether the received file search instruction is a structured conditional search instruction. If it is determined that the file search instruction is a structured conditional search instruction, it can further decompose the search conditions of the structured conditional search instruction, that is, perform task decomposition, and combine asynchronous pipelines and lock-free communication to achieve parallel search optimization and improve search efficiency.

[0189] It should be noted that the search engine module provides a unified and structured search method and is responsible for efficiently processing search tasks; it may specifically include a unified interface access unit, a structured condition processing unit and a parallel search execution unit.

[0190] The unified interface access unit is used to receive file retrieval instructions through various types of standardized interfaces.

[0191] The structured condition processing unit is used to process the file retrieval instructions input by the user into multiple retrieval tasks that support parallel retrieval. Each retrieval task is a task of searching based on the corresponding retrieval conditions. Specifically, it can analyze and process the structured condition retrieval instructions input by the user to generate a retrieval plan, optimize the retrieval strategy, and select at least two target index libraries to execute the retrieval task; thereby generating a target retrieval strategy.

[0192] The parallel search execution unit is used to schedule multiple back-end data sources in parallel to collaboratively search for file metadata and merge multi-source data; that is, through task decomposition, asynchronous pipelines, and lock-free communication, it coordinates multiple index data sources in multiple search plans to achieve efficient parallel collaboration.

[0193] In this embodiment, examples of all conditions supported by the structured condition processing unit are as follows:

[0194] 1) Metadata indicators, such as file name, file name of the soft link associated with the file, path, file type, file size, file owner, file group, file permissions, last access time, last modification time, last status change time, and file creation time;

[0195] 2) Comparison conditions: not equal to, greater than, less than, greater than or equal to, less than or equal to;

[0196] 3) Logical conditions: satisfying multiple conditions at the same time, satisfying any one condition, or not satisfying a condition;

[0197] 4) Range conditions: within a certain range, not within a certain range;

[0198] 5) Fuzzy matching conditions, using the wildcard * to match strings;

[0199] 6) Regular expression conditions: use regular expressions to process conditional retrieval of file names and file paths.

[0200] Furthermore, the parallel search execution unit executes the target search strategy and concurrently schedules the index data sources of at least two target index libraries to collaboratively search for file metadata until a file search result is obtained.

[0201] Exemplarily, the retrieval process of the parallel retrieval execution unit is as follows:

[0202] By parsing the search conditions for the file search instruction "Find Word files modified in the last week", we know that we need to search for files with extensions such as doc, docx, odt, etc., and whose last modification time is within a week. According to the parsed search conditions, the updated inverted index library and the updated metadata index library need to be used as two target index libraries for collaborative search.

[0203] First, for the first search plan, specifically in the updated inverted index library, which is the target index library, a search is performed using doc, docx, and odt as keywords, and all retrieved file IDs are placed in a lock-free circular queue.

[0204] Then, for the second retrieval plan, all file IDs are taken out from the lock-free circular queue, and the corresponding file data is retrieved in the updated metadata index library, which is the target index library, and conditional filtering is performed based on the last modification time, file type and file name suffix.

[0205] Finally, the file data after condition filtering is sorted in descending order according to the last modification time, and the file data after descending order is fed back to the visual human-computer interaction interface as the file retrieval result.

[0206] It should be noted here that in order to improve retrieval efficiency and reduce retrieval resources, the file data set that matches the file retrieval instruction can be first retrieved in the updated cache space; the updated cache space is obtained by updating the existing file ID-cumulative retrieval frequency-retrieval priority mapping relationship table in the cache space based on the file ID of each target file.

[0207] If the file dataset exists in the updated cache space, there is no need to perform collaborative search in the two target index libraries mentioned above, and this file dataset is determined as the file retrieval result; conversely, if the file dataset does not exist in the updated cache space, the file retrieval result is obtained by using the collaborative search method of the two target index libraries mentioned above.

[0208] Based on the above Figure 1 In the method shown, in an exemplary embodiment, the user state space can also process the unstructured conditional search instruction input by the user into a structured conditional search instruction, and the specific processing process can be implemented through the following steps in this embodiment.

[0209] In the case where the file retrieval instruction is a natural language retrieval instruction, the natural language retrieval instruction is processed into a structured language retrieval instruction based on a pre-trained target language processing model.

[0210] Among them, the target processing model is obtained by training the NLP model based on the Transformer framework based on sample natural language retrieval instructions in different file retrieval scenarios and the sample structured retrieval instructions identified by each sample natural language retrieval instruction; and each sample natural language retrieval instruction has undergone data enhancement.

[0211] It should be noted that the present invention converts natural language into structured conditions by introducing an NLP model based on the Transformer architecture, realizes file retrieval based on natural language, and integrates it with the traditional structured condition retrieval method to support multi-mode hybrid retrieval.

[0212] Specifically, the user space can also include a natural language processing module, introducing an NLP model based on the Transformer architecture. This NLP model is trained to automatically process input natural language search commands into structured language search commands. This automatically identifies semantic elements, achieving end-to-end conversion from natural language and find command syntax to structured search conditions.

[0213] For the NLP model based on the Transformer architecture, its training process is carried out by using a natural language corpus and a find command corpus, realizing end-to-end conversion from natural language to structured retrieval conditions.

[0214] The natural language corpus is constructed by collecting sample natural language search commands from various file retrieval scenarios and annotating them with corresponding sample structured search commands. This covers scenarios such as operations and maintenance, development, and user document management, and is expanded to include logical searches. Data augmentation (i.e., noise injection) is used to improve model generalization. Furthermore, the corresponding find search command for each natural language search command is identified with the corresponding structured search condition as the find command for training. This yields a training sample set, in which each training sample is a sample natural language search command from the corresponding file retrieval scenario, and this sample natural language search command has undergone data augmentation and is labeled with a sample structured search command.

[0215] The natural language processing module can automatically identify semantic elements and includes a preprocessing unit, an encoding unit, a decoding unit, a vocabulary mapping unit and an agent retrieval unit.

[0216] The preprocessing unit is used to convert the target text instructions into a token sequence based on the tokenizer, and then convert the token sequence into a feature vector.

[0217] The encoding unit is used to embed the feature vector output by the preprocessing unit into a high-dimensional vector space and add positional encoding to preserve the sequence order information, thereby obtaining a latent vector that can abstractly represent the token sequence. This latent vector can be used as a compressed, summarized, and abstract expression of the target text instruction, facilitating the model's subsequent information processing and extraction of more features.

[0218] Subsequently, all high-level semantic features in the latent vector are extracted again through the multi-head self-attention mechanism, that is, the relationship between all high-level semantic features in the latent vector is analyzed, and a more detailed and refined feature representation vector is generated based on all the analyzed relationships. This abstract and refined feature representation vector will then be used by the decoding unit to generate target language instructions.

[0219] The decoding unit is used to convert the fine feature representation vector generated by the encoding unit into an output token sequence, mapping the features of the intermediate latent state space to the desired target space. The decoding unit is an autoregressive architecture that gradually builds structured retrieval conditions through the autoregressive mechanism. The decoder predicts only one token in the output token sequence at a time and outputs the prediction result. After outputting the prediction result for the current token, it uses this prediction result as the input for the next prediction and feeds it into the decoder along with the fine feature representation vector output by the encoder. Because the vector output by the decoder already contains the contextual connection between each token and other tokens, when using the decoder for prediction, only the current token to be predicted and the previous prediction results can be used as input.

[0220] The vocabulary mapping unit converts the output token sequence generated by decoding into a structured language retrieval instruction, which includes two stages: field mapping and logical mapping. The field mapping language translates and converts each retrieval field and its own constraints, and the logical mapping is used to translate and convert the logical relationship between multiple retrieval fields. After two-stage mapping, a structured language retrieval instruction combining conditions and logic is formed, and the request is further forwarded through the proxy retrieval unit.

[0221] For example, for the structured language search instruction "find Word files modified in the last week", the keywords to be parsed include last week, modification and Word file, and the search time range needs to be determined, and the search time parameter is the modification time of the file meta-attribute; for Word, the mapping extension name is files in formats such as doc, docx, odt, etc.

[0222] In this embodiment, the structured language search condition example converted from the above natural language search instruction example is described as follows:

[0223] ```

[0224] {

[0225] "path": [" / "],

[0226] "constraints": [

[0227] {

[0228] "id": 0,

[0229] "field": "mtime",

[0230] "operator": "ge",

[0231] "value": timestamp(s)

[0232] },

[0233] {

[0234] "id": 1,

[0235] "field": "suffix",

[0236] "operator": "in",

[0237] "value": ["doc", "docx", "odt"]

[0238] },

[0239] {

[0240] "id": 2,

[0241] "field": "type",

[0242] "operator": "eq",

[0243] "value": "file"

[0244] }

[0245] ],

[0246] "logic": [

[0247] {

[0248] "type": "and",

[0249] "constraints": [0, 1, 2]

[0250] } ]

[0252] }

[0253] ```

[0254] Among them, path indicates the directory range limited by the search, and the search starts from the root directory under the default conditions; constraints indicates the search-related constraints, which supports all conditions supported by all structured condition processing units; id indicates the unique number of each constraint; field indicates the type of constraint, such as mtime constraining the file modification time; operator indicates some search operation restrictions specified for the constraint, such as ge indicating the operation behavior greater than or equal to; value indicates the target value that the constraint needs to meet, such as the search time specified by the timestamp with an accuracy of seconds; logic indicates the logical relationship between multiple constraints during the search; type indicates the logical type, which can be and or or; constraints indicates the related constraints participating in the logical operation.

[0255] The proxy retrieval unit is used to further encapsulate the structured language retrieval instructions output by the vocabulary mapping unit according to the interface standard of the retrieval engine module, forward the corresponding retrieval request, and finally return the file retrieval result.

[0256] It should be noted that the above description of the four units, namely the preprocessing unit, the encoding unit, the decoding unit and the vocabulary mapping unit, can be used in the model training stage and in the subsequent practical application stage; for the model training stage, the target text instruction can specifically be a sample natural language retrieval instruction in different file retrieval scenarios and each sample natural language retrieval instruction has been data enhanced and the corresponding sample structured retrieval instruction is identified; for the subsequent practical application stage, the target text instruction can specifically be the natural language retrieval instruction input by the user at the current moment. On the trained target language processing model, the natural language retrieval instruction is converted into a structured retrieval instruction, and then a retrieval request is further initiated to the retrieval engine module through proxy retrieval. That is, the principles of the model training stage and the practical application stage are the same. Since the present invention does not improve the structure of the NLP model itself based on the Transformer framework, and the improvement lies in the collection of training sample sets and training it to have the function of converting natural language into structured language, here we only provide an overview of the model training process and the subsequent practical application process in combination with the training sample sets.

[0257] The embodiment of the present invention introduces an NLP model based on the Transformer architecture to convert natural language into structured language, realizes file retrieval based on natural language, and integrates it with the traditional structured condition retrieval method to support multi-mode hybrid retrieval, thereby improving the flexibility and accuracy of file retrieval.

[0258] Based on the above Figure 1In the method shown in FIG. 1 , in an exemplary embodiment, the target search strategy corresponding to the file search instruction is determined in step 101. The specific determination process in this embodiment can be performed by Figure 5 Steps 501 to 502 are implemented as shown.

[0259] Step 501: parse the file retrieval instruction for retrieval conditions, and determine a target retrieval plan based on all parsed retrieval conditions.

[0260] Step 502: Based on the preset mapping relationship between the retrieval plan and the retrieval strategy, determine the target retrieval strategy corresponding to the target retrieval plan.

[0261] Specifically, keywords are extracted from the file search instruction, and search conditions are determined based on the metadata fields mapped to all the extracted keywords. That is, when all the extracted keywords include metadata fields, all the keywords can be used as search conditions.

[0262] For example, for the file search instruction "Find Word files modified in the last week", all the keywords extracted include word file, last week and modification. Word file maps to files with extensions such as doc, docx, odt, etc. Last week and modification map to the last modification time within a week, thus obtaining two search conditions.

[0263] At this time, target search plans corresponding to all search conditions may be determined. The number of target search plans may be the same as the number of search conditions and correspond one to one. Each target search plan includes the corresponding search condition and a processing method for the search result.

[0264] Furthermore, based on the pre-set mapping relationship between the retrieval plan and the retrieval strategy, a target retrieval strategy corresponding to the target retrieval plan may be determined; the target retrieval strategy specifically refers to a specific implementation process for realizing the corresponding target retrieval plan.

[0265] Exemplarily, when the target retrieval plan includes a first retrieval plan and a second retrieval plan, the target retrieval strategy corresponding to the first retrieval plan is to perform retrieval based on corresponding retrieval conditions and place the retrieved candidate file IDs into a lock-free circular queue; the target retrieval strategy corresponding to the second retrieval plan is to take out all candidate file IDs from the lock-free circular queue, and retrieve the file data corresponding to each candidate file ID, and then further perform conditional filtering on all retrieved file data based on the corresponding retrieval conditions, and then other processing such as descending or ascending sorting can be performed on the conditional filtering results.

[0266] The file retrieval method provided by the embodiment of the present invention utilizes a kernel-mode event-driven zero-copy data acquisition mechanism, a user-mode dynamic hybrid indexing architecture, and natural language semantic parsing technology. Specifically, through the collaborative architecture of kernel-mode event-driven and user-mode hybrid indexing, file operation events are directly monitored in the kernel mode, and the index database is updated in real time. A dynamic sharding queue architecture is proposed in the user mode to ensure the order and performance of the hybrid index during concurrent event updates, supporting millisecond-level responses. Natural language retrieval is implemented by parsing natural language into a structured retrieval language through an NLP engine, enabling end users to complete complex file retrieval in a daily conversational manner with zero technical barriers, thereby achieving real-time updates of file system metadata, efficient retrieval, and natural language interaction capabilities.

[0267] It should be noted that although the operations of the present method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0268] In one embodiment, the present invention further provides a file retrieval device, such as Figure 6 As shown, the file retrieval device includes: a policy determination unit 601, an event acquisition unit 602, an index library updating unit 603 and a file retrieval unit 604.

[0269] The policy determination unit 601 is configured to determine the target search policy corresponding to the file search instruction in response to the file search instruction currently received in the user state space, and update the event filtering policy to the kernel state space.

[0270] The event acquisition unit 602 is configured to acquire a target file operation event from all file operation events captured at the current moment in the kernel space based on the updated event filtering policy.

[0271] The index library update unit 603 is configured to determine the metadata information of the target file corresponding to the target file operation event in the user space, and update the hybrid index database based on the metadata information; the updated hybrid index database includes the mapping relationship between the target file and the metadata information, and the mapping relationship between the historical file and the historical metadata information; and the hybrid index database supports multiple types of retrieval conditions.

[0272] The file retrieval unit 604 is configured to perform a search in the updated hybrid index database based on the target retrieval strategy and the file retrieval instruction in the user space to obtain a file retrieval result.

[0273] In one embodiment, the strategy determination unit 601 is specifically configured to parse the retrieval conditions of the file retrieval instruction and determine the target retrieval plan based on all the parsed retrieval conditions; based on the pre-set mapping relationship between the retrieval plan and the retrieval strategy, determine the target retrieval strategy corresponding to the target retrieval plan.

[0274] In one embodiment, the policy determination unit 601 is specifically configured to obtain a preset format rule configuration file at the current moment; parse the event filtering rules from the preset format rule configuration file, and perform a validity check on the event filtering rules; and use the event filtering rules that pass the validity check as updated event filtering policies, and update them in real time to the kernel state space.

[0275] In one embodiment, the event acquisition unit 602 is specifically configured to perform different levels of event filtering on all file operation events based on the different levels of event filtering policies included in the updated event filtering policies, so as to filter out target file operation events from all file operation events; wherein, the different levels of event filtering policies include a first-level event filtering policy based on a preset path prefix filtering, and a second-level event filtering policy based on a preset file type and a preset file suffix filtering.

[0276] In one embodiment, the index library update unit 603 is specifically configured to update the existing keyword-file ID mapping relationship table in the inverted index library based on the file ID and file name keyword of the target file when the hybrid index database includes an inverted index library, a prefix tree index library and a metadata index library; update the existing file path-file ID mapping relationship table in the prefix tree index library based on the file wildcard and file ID of the target file and the file path in the metadata information; and update the existing metadata information-file ID mapping relationship table in the metadata index library based on the metadata information and the file ID.

[0277] In one embodiment, the file retrieval unit 604 is specifically configured to determine at least two target index libraries in the updated inverted index library, the updated prefix tree index library, and the updated metadata index library based on the retrieval conditions corresponding to the file retrieval instruction; based on the target retrieval strategy, perform collaborative retrieval in the at least two target index libraries to obtain file retrieval results.

[0278] In one embodiment, the strategy determination unit 601 is further specifically configured to, when the file retrieval instruction is a natural language retrieval instruction, process the natural language retrieval instruction into a structured language retrieval instruction based on a pre-trained target language processing model.

[0279] Among them, the target processing model is obtained by training the NLP model based on the Transformer framework based on sample natural language retrieval instructions in different file retrieval scenarios and the sample structured retrieval instructions identified by each sample natural language retrieval instruction; and each sample natural language retrieval instruction has undergone data enhancement.

[0280] It should be understood that the units recorded in the document retrieval device and the reference Figure 1 The various steps in the described method correspond to each other. Therefore, the operations and features described above for the method are also applicable to the file retrieval device and the units included therein, and will not be repeated here. The file retrieval device can be pre-implemented in a browser or other security application of a computer device, or can be loaded into the browser or its security application of a computer device through downloading or other means. The corresponding units in the file retrieval device can cooperate with the units in the computer device to implement the solutions of the embodiments of the present invention.

[0281] Reference below Figure 7 , which shows a schematic structural diagram of a computer system 700 of a terminal device or server suitable for implementing an embodiment of the present invention.

[0282] like Figure 7 As shown, computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of system 700 are also stored in RAM 703. CPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0283] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 708 including devices such as a hard disk; and a communication section 709 including a network interface card such as a LAN card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. Removable media 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read from the media can be installed in the storage section 708 as needed.

[0284] In particular, according to the embodiments of the present disclosure, the above reference Figure 1The described processes may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising instructions for executing Figure 1 In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 709 and / or installed from the removable medium 711 .

[0285] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, or any suitable combination thereof.

[0286] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0287] The units or modules described in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be provided in a processor. The names of these units or modules do not, in certain circumstances, limit the units or modules themselves.

[0288] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the computer device described in the above embodiment, or may exist independently without being assembled into the computer device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the method of the present invention. For example, it can be executed Figure 1 The individual steps of the method are shown.

[0289] An embodiment of the present invention provides a computer program product, which includes instructions. When the instructions are executed, the method described in the embodiment of the present invention is executed. For example, Figure 1 The individual steps of the method are shown.

[0290] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, database, or other media used in the embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, and the like.

[0291] The above description is merely a preferred embodiment of the present invention and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in the present invention.

Claims

1. A file retrieval method, characterized in that: Applied to a computer device, the computer device is installed with an operating system, the operating system includes a user state space and a kernel state space; the method includes: The user state space determines a target search strategy corresponding to the file search instruction in response to the file search instruction received at the current moment, and updates the event filtering strategy to the kernel state space; The kernel state space obtains a target file operation event from all file operation events captured at the current moment based on the updated event filtering policy; all file operation events include file creation events, file deletion events, file rename events, file write events, file permission modification events, file owner modification events, file soft link creation events, and file hard link creation events, and all file operation events matching pre-configured kernel key functions are captured by injecting an eBPF program into the system call layer and the VFS layer of the kernel state space to dynamically hook the pre-configured kernel key functions; The user space determines metadata information of the target file corresponding to the target file operation event, and updates the hybrid index database based on the metadata information; the updated hybrid index database includes a mapping relationship between the target file and the metadata information, and a mapping relationship between historical files and historical metadata information; and the hybrid index database supports multiple types of search conditions; The user space searches the updated hybrid index database based on the target search strategy and the file search instruction to obtain a file search result; The step of obtaining a target file operation event from all file operation events captured at the current moment based on the updated event filtering policy includes: Based on the different levels of event filtering policies included in the updated event filtering policy, performing different levels of event filtering on all the file operation events to filter out the target file operation event from all the file operation events, where the target file operation event is a file operation event that is not successfully matched by at least one level of filtering in the two-level filtering; The event filtering strategies of different levels include a first-level event filtering strategy based on filtering based on a preset path prefix, and a second-level event filtering strategy based on filtering based on a preset file type and a preset file suffix.

2. The method according to claim 1, characterized in that The updating of the hybrid index database based on the metadata information includes: In the case where the hybrid index database includes an inverted index library, a prefix tree index library and a metadata index library, Based on the file ID and file name keywords of the target file, updating the existing keyword-file ID mapping relationship table in the inverted index library; Based on the file ID of the target file and the file path in the metadata information, updating the existing file path-file ID mapping relationship table in the prefix tree index library; Based on the metadata information and the file ID, the existing metadata information-file ID mapping relationship table in the metadata index library is updated.

3. The method according to claim 2, characterized in that The step of performing a search in the updated hybrid index database based on the target search strategy and the file search instruction to obtain a file search result includes: Based on the search condition corresponding to the file search instruction, determining at least two target index libraries from the updated inverted index library, the updated prefix tree index library, and the updated metadata index library; Based on the target retrieval strategy, a collaborative search is performed in the at least two target index libraries to obtain the document retrieval result.

4. The method according to any one of claims 1 to 3, characterized in that The updating of the event filtering policy to the kernel state space includes: Obtaining the preset format rule configuration file at the current moment; Parsing the event filtering rules from the preset format rule configuration file and performing a validity check on the event filtering rules; The event filtering rules that pass the legality check are used as updated event filtering strategies and updated to the kernel state space in real time.

5. The method according to any one of claims 1 to 3, characterized in that The method further comprises: In the case where the file retrieval instruction is a natural language retrieval instruction, processing the natural language retrieval instruction into a structured language retrieval instruction based on a pre-trained target language processing model; Among them, the target processing model is obtained by training the NLP model based on the Transformer framework based on sample natural language retrieval instructions in different file retrieval scenarios and sample structured retrieval instructions identified by each sample natural language retrieval instruction; and each of the sample natural language retrieval instructions has undergone data enhancement.

6. The method according to any one of claims 1 to 3, characterized in that Determining the target retrieval strategy corresponding to the file retrieval instruction includes: Parsing the search conditions of the file search instruction and determining a target search plan based on all the parsed search conditions; Based on a preset mapping relationship between the retrieval plan and the retrieval strategy, the target retrieval strategy corresponding to the target retrieval plan is determined.

7. A file retrieval device, characterized in that: The device comprises: a policy determination unit configured to determine, in response to a file retrieval instruction currently received in the user state space, a target retrieval policy corresponding to the file retrieval instruction, and update the event filtering policy to the kernel state space; an event acquisition unit configured to acquire, in a kernel-mode space, a target file operation event from all file operation events captured at the current moment in response to an updated event filtering policy; wherein all file operation events include file creation events, file deletion events, file rename events, file write events, file permission modification events, file owner modification events, file soft link creation events, and file hard link creation events, and are all file operation events that match pre-configured kernel key functions and are captured by dynamically hooking pre-configured kernel key functions by injecting an eBPF program into the system call layer and the VFS layer of the kernel-mode space; An index library update unit configured to determine metadata information of a target file corresponding to the target file operation event in the user space and update a hybrid index database based on the metadata information; the updated hybrid index database includes a mapping relationship between the target file and the metadata information, and a mapping relationship between historical files and historical metadata information; and the hybrid index database supports multiple types of search conditions; A file retrieval unit configured to perform a search in the updated hybrid index database in the user space based on the target retrieval strategy and the file retrieval instruction to obtain a file retrieval result; The step of obtaining a target file operation event from all file operation events captured at the current moment based on the updated event filtering policy includes: Based on the different levels of event filtering policies included in the updated event filtering policy, performing different levels of event filtering on all the file operation events to filter out the target file operation event from all the file operation events, where the target file operation event is a file operation event that is not successfully matched by at least one level of filtering in the two-level filtering; The event filtering strategies of different levels include a first-level event filtering strategy based on filtering based on a preset path prefix, and a second-level event filtering strategy based on filtering based on a preset file type and a preset file suffix.

8. A computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the file retrieval method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the file retrieval method according to any one of claims 1 to 6 is implemented.

Images