Index processing method and device for file system, file processing method and device, storage medium and computer program product

Abstract

The embodiment of the invention provides an index processing method and device for a file system, a file processing method and device, a storage medium and a computer program product. In the embodiment of the invention, when the index table of the file system is created, the index logic adopts ordered table storage, each row is isomorphic, and the skip table pointer is embedded, so that the constructed index table is high in storage efficiency, supports efficient sequential access and is high in random access efficiency. In addition, the index table of the file system is created only by traversing the file system once, the I / O frequency is reduced to the maximum extent, I / O reading and writing are based on sequential reading and writing, and the creation efficiency is high. The index table of the file system provides rich file information, file retrieval under complex conditions can be supported, file retrieval efficiency is high, and resource overhead is low.

Description

Technical Field

[0001] The present application relates to the technical field of file systems, and in particular to a file system index processing method, file processing method, device, storage medium, and computer program product. Background Art

[0002] A file system is the method and data structure used by an operating system to identify files on a storage device or partition. With the continuous development of big data and cloud computing technologies, the demand for high-performance file system access is becoming increasingly urgent. File system indexes enable efficient file retrieval within the file system, reducing resource overhead. Therefore, file system indexing schemes have been a hot topic of research. Summary of the Invention

[0003] Multiple aspects of the present application provide a file system index processing method, a file processing method, a device, a storage medium, and a computer program product, so as to provide a better file system indexing solution.

[0004] An embodiment of the present application provides an index processing method for a file system, comprising: traversing a directory tree of the file system using a depth-first search method to obtain metadata and topological structure information of multiple files in the file system; recording file information of the multiple files in a current index table in sequence according to the traversal order of the multiple files, the file information including the metadata and topological structure information of the files; obtaining a skip list of the non-empty directory files based on the file information of the subfiles of the non-empty directory files in the directory tree; and adding a next pointer of the subfile in the skip list of the non-empty directory file to the row of the subfile in the current index table to complete the creation of the current index table, wherein the next pointer points to the next node of the same-level linked list.

[0005] An embodiment of the present application provides a file processing method, including: accessing the current index table of the file system to read the file information of each file in the current index table; filtering the file information of each read file according to the retrieval condition to obtain the target file information that meets the retrieval condition; and retrieving the target file corresponding to the target file information in the file system.

[0006] An embodiment of the present application provides a file detection method, characterized in that it includes: traversing the current index table of the file system from top to bottom; if the file information of the directory file is currently traversed, then when the number of sub-files of the directory file is greater than a set threshold, judging whether the current remaining resource amount of the resource pool meets the resource demand for sub-task execution; if it is satisfied, splitting out a sub-index table from the current index table, the sub-index table including the file information of the directory file and its sub-files; creating a sub-task based on the sub-index table, allocating resources in the resource pool to the sub-task, and executing the sub-task based on the allocated resources to obtain the detection result of the sub-task, and returning the allocated resources to the resource pool after the sub-task is executed; merging the detection results of each sub-task to obtain a final detection result.

[0007] An embodiment of the present application provides a computer device, comprising: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program to execute the steps in the index processing method, file processing method or file detection method of the file system.

[0008] An embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the index processing method or file processing method of the file system.

[0009] An embodiment of the present application provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement the steps in the index processing method, file processing method or file detection method of the file system.

[0010] In an embodiment of the present application, when creating an index table for a file system, the index logic uses ordered table storage, isomorphic rows, and an embedded skip table pointer. The index table constructed in this way has high storage efficiency and supports efficient sequential access and random access. In addition, creating an index table for a file system only requires traversing the file system once, which minimizes the number of I / O (Input / Output) operations, and the read and write I / O is mainly sequential, resulting in high creation efficiency. The index table of the file system provides rich file information and can support file retrieval under complex and compound conditions, with high file retrieval efficiency and low resource overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1A flowchart of a file system index processing method provided in an embodiment of the present application;

[0013] Figure 2 The data structure of the exemplary index table provided in the embodiment of the present application;

[0014] Figure 3 A schematic diagram of an exemplary sequential access index table;

[0015] Figure 4 A schematic diagram of an exemplary sequential access index table;

[0016] Figure 5 is a schematic diagram of an exemplary random access index table;

[0017] Figure 6 is a schematic diagram of comparison between exemplary index tables;

[0018] Figure 7 is a schematic diagram of comparison between exemplary index tables;

[0019] Figure 8 A schematic diagram of an exemplary index table splitting;

[0020] Figure 9 An expanded schematic diagram of an exemplary index table;

[0021] Figure 10 is a schematic diagram of the structure of an exemplary index page;

[0022] Figure 11 This is a schematic diagram of an exemplary index file loading process;

[0023] Figure 12 A flowchart of a file retrieval method based on a file system index provided in an embodiment of the present application;

[0024] Figure 13 is an exemplary diagram of a document retrieval process;

[0025] Figure 14 is an exemplary diagram of a file detection process;

[0026] Figure 15 A schematic diagram of the structure of an index processing device for a file system provided in an embodiment of the present application;

[0027] Figure 16 A schematic diagram of the structure of a file processing device provided in an embodiment of the present application;

[0028] Figure 17 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the access relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of the present application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship. In addition, in the embodiments of the present application, "first", "second", "third", etc. are only used to distinguish the contents of different objects and have no other special meanings.

[0031] A file system is the method and data structure used by an operating system to identify files on a storage device or partition. With the continuous development of big data and cloud computing technologies, the demand for high-performance file system access is becoming increasingly urgent. File system indexes enable efficient file retrieval within the file system, reducing resource overhead. Therefore, file system indexing schemes have been a hot topic of research.

[0032] To this end, an embodiment of the present application provides an index processing method, a file processing method, a device, a storage medium and a computer program product for a file system. In an embodiment of the present application, when creating an index table of a file system, the index logic adopts ordered table storage, isomorphism of each row, and an embedded skip table pointer, so that the index table constructed in this way has high storage efficiency and supports efficient sequential access and random access with high efficiency. In addition, creating an index table of a file system only requires traversing the file system once, which minimizes the number of I / O (Input / Output) times, and the read and write I / O is mainly sequential read and write, and the creation efficiency is high. The index table of the file system provides rich file information, can support file retrieval with complex compound conditions, and has high file retrieval efficiency and low resource overhead.

[0033] First, some terms involved in the embodiments of this application are briefly introduced.

[0034] Depth-First Search (DFS) algorithm: It is a graph algorithm that starts from the starting vertex and selects a path to deeply explore the target vertex. If the target vertex does not exist on the path, it backtracks to the starting vertex and continues to select another path to deeply explore the target vertex until the target vertex is found or all vertices are explored before backtracking to the starting vertex to complete the search.

[0035] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0036] Figure 1 This is a flowchart of a file system index processing method provided by an embodiment of the present application. Figure 1 , the method may include the following steps:

[0037] 101. Use a depth-first search method to traverse the directory tree of the file system to obtain metadata and topological structure information of multiple files in the file system;

[0038] 102. Record file information of the multiple files in the current index table in sequence according to the traversal order of the multiple files, where the file information includes metadata and topology information of the files;

[0039] 103. Obtain a jump list for the non-empty directory file based on the file information of the subfiles of the non-empty directory file in the directory tree;

[0040] 104. Add the next pointer of the sub-file in the jump list of the non-empty directory file to the row of the sub-file in the current index table to complete the creation of the current index table, wherein the next pointer points to the next node of the same-level linked list.

[0041] Specifically, a file system includes multiple files, including directory files (also referred to as directories) and non-directory files. Directory files are further divided into empty directory files (also referred to as empty directories) and non-empty directory files (also referred to as non-empty directories). A non-empty directory is a directory that includes information about the next-level subdirectories or non-directory files, while an empty directory is a directory that does not include information about the next-level subdirectories or non-directory files. Non-directory files are files other than directory files in the multiple files. Examples of non-directory files include, but are not limited to, ordinary files, symbolic link files, device files, socket files, queue (FIFO) files, symbolic link files, and the like. Examples of ordinary files include, but are not limited to, text files, data files, binary files, and the like. Examples of device files include, but are not limited to, block device files and character device files.

[0042] In this embodiment, the directory tree of a file system is a tree structure that describes the hierarchical relationships between files in the file system. Each node in the directory tree corresponds to a file and includes metadata about the file. Optionally, the file metadata included in the node in the directory tree includes, but is not limited to, the file type (type), file size (size), file owner (owner), file creation time (ctime), file last access time (atime), and file last modification time (mtime).

[0043] Among them, file type (type): identifies the type of a file. Common file types include directory files, ordinary files, socket files, queue (FIFO) files, symbolic link files, etc.

[0044] File size: refers to the size of the disk space occupied by the file.

[0045] The file owner is the person who owns the file, usually the creator of the file.

[0046] File creation time (ctime): refers to the time when the file was created. The file creation time can be the difference from midnight on January 1, 1970, in seconds.

[0047] File last access time (atime): refers to the time when the file was last accessed. The file last access time can be the difference from midnight on January 1, 1970, in seconds.

[0048] File last modification time (mtime): refers to the time when the file was last modified. The file last modification time can be the difference from midnight on January 1, 1970, in seconds.

[0049] In this embodiment, a depth-first search method is adopted to traverse the directory tree of the file system, wherein multiple child nodes of a parent node in the directory tree are traversed in order of file names.

[0050] Specifically, a depth-first search is used to traverse the directory tree from the root node down, prioritizing the children of the previous node and then the siblings. Multiple children of the parent node in the directory tree are traversed in file name order. For the currently traversed node, metadata for the file corresponding to the current node is obtained. This metadata may not include the file path.

[0051] In this embodiment, there is no restriction on the file name sorting method. For example, the file names may be sorted in dictionary order of the first character in the file name. Another example is sorting based on the number of characters in the file name. Another example is sorting based on the number of words in the file name.

[0052] In this embodiment, metadata of multiple files are recorded in the current index table in sequence according to the traversal order of the multiple files. Furthermore, the row number of the row storing the file information of the file in the current index table is consistent with the traversal sequence number of the file. The earlier the node in the traversal order, the smaller the row number in the index table.

[0053] by Figure 2 For example, the nodes corresponding to directory files on the directory tree are home node, cai node, root node, admin node, logs node, etc. The nodes corresponding to non-directory files on the directory tree are note.txt node, readme.md node, sound.mp3 node, pub.key node, 1220.log node, etc. Traverse the directory tree in depth-first search mode, and traverse the multiple child nodes of the parent node in lexicographic order of file names. Figure 2 The order of traversal of the nodes in the directory tree shown is: home, admin, logs, 1220.log, cai, note.txt, readme.md, root, pub.key, and sound.mp3. The index table records the file metadata corresponding to each node. Nodes that are earlier in the traversal order have smaller row numbers in the index table. The row numbers corresponding to the nodes in the traversal order are 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9, respectively.

[0054] In the process of traversing the directory tree, in addition to obtaining the file's metadata, you can also obtain the file's topological structure information. The file's topological structure information is added to the corresponding row in the current index table. The file's topological structure information reflects the parent-child relationship between the file's node and other file nodes in the directory tree.

[0055] In this embodiment, in addition to recording the metadata of the file in the current index table, the topological structure information of the file is also recorded in the current index table. The topological structure information of the file includes, but is not limited to, the file name (File Name), the number of subfiles (Subfile Number), or the parent pointer (Parent).

[0056] Among them, the file name is the unique identifier of the file. The number of subfiles (subfile num) refers to the number of all files corresponding to the next-level subnode of the node on the directory tree. For non-directory files, the number of subfiles is 0; for directory files, the number of subfiles is the number of all files corresponding to the next-level subnode; the parent pointer (parent) reflects the difference between the row number corresponding to the child node on the directory tree and the row number corresponding to its parent node; for the row in the index table that records the file information of any file, based on the row number and parent pointer of the row, the row number of the parent file of the file in the index table is obtained. The parent file of the file is also the file corresponding to the parent node of the node of the file on the directory tree.

[0057] Continue with Figure 2 For example, the home node, admin node, logs node, root node and other nodes in the directory tree have child nodes, and the admin node, logs node, 1220.log node, cai node, note.txt node, readme.md node, root node, pub.key node and sound.mp3 node and other nodes have parent nodes. Based on the number of child files and the definition of parent pointer, the topological structure information of the file corresponding to each node is determined, and the topological structure information of the file is recorded in the row where the node is located in the index table. Figure 2 The parent pointer is NUL, which means there is no parent node. Therefore, the index table records more abundant file information.

[0058] In this embodiment, for any non-empty directory file among the multiple files, the relevant pointer of the jump table of the non-empty directory file is added to the row corresponding to the non-empty directory file in the current index table to complete the creation of the current index table.

[0059] Specifically, a skip list is an ordered data structure based on a linked list that supports efficient insertion, deletion, and querying. A node in a skip list contains two pointers: one pointer points to the next node in the linked list at the same level, and the other pointer points to the node with the same element in the linked list below.

[0060] In this embodiment, the file information of the sub-file corresponding to the next-level child node of the node to which the non-empty directory file belongs in the directory tree forms a jump list of the non-empty directory file, that is, the nodes in the jump list of the non-empty directory file correspond to the next-level child node of the node to which the non-empty directory file belongs, and the arrangement order of the nodes in the jump list of the non-empty directory file conforms to the arrangement order of the file names.

[0061] In practical applications, there is no limit on the number of levels of the skip list for non-empty directory files. Preferentially, the number of levels of the skip list for non-empty directory files is 3, that is, from top to bottom, the skip list for non-empty directory files includes the third-level linked list, the second-level linked list, and the first-level linked list in sequence; the node span of the first-level linked list is 1, that is, the first-level linked list includes all sub-files under the non-empty directory file; the node span of the second-level linked list is the maximum value of 2 and n / 8, and the node span of the second-level linked list is the maximum value of 4 and n / 4, where n is the number of sub-files under the non-empty directory file. The node span can be understood as the distance between two nodes in the skip list.

[0062] In this embodiment, when constructing a jump list for a non-empty directory file, for each node in the linked list, it is necessary to determine the next pointer and down pointer of the node. The next pointer points to the next node in the linked list at the same level, and the down pointer points to the node with the same element in the linked list at the lower level. Figure 2 For example, the index table records jump table pointers such as next1, next2, and next3. Next1 is the next pointer of the node in the first-level linked list, next2 is the next pointer of the node in the second-level linked list, and next3 is the next pointer of the node in the third-level linked list. Figure 2 The pointer value of the next pointer is NUL, which means that there is no next node. The pointer value of the next pointer is 1, 2, 3, 4, 6, etc., which means that the difference in row numbers between adjacent nodes in the same-level linked list is 1, 2, 3, 4, 6, etc. It is worth noting that for empty directory files or non-directory files, the pointer value of the next pointer corresponding to these files can be set to NUL in the index table, which means that these files have no child files.

[0063] In this embodiment, the next pointer of the sub-file of the non-empty directory file in the jump list of the non-empty directory file is added to the row where the corresponding sub-file is located in the current index table, so as to complete the creation of the current index table.

[0064] In this embodiment, the created index table has the following advantages:

[0065] 1. High storage efficiency. It stores file names instead of full paths, saving space wasted on storing a large number of identical path prefixes.

[0066] Specifically, because file paths within the same directory all contain the same prefix, and most file systems have no restrictions on file tree depth or file path length, indexing file paths would waste significant storage resources. Therefore, the index table stores only the file name in each row, saving space and conserving storage resources. Furthermore, traversing the index table allows for dynamic file path generation, enabling file retrieval based on the file path.

[0067] 2. Each row in the index table is homogeneous, which facilitates random access to the index table based on the row number.

[0068] 3. The index table has a built-in jump table pointer, which can quickly locate the row in the index table based on the file path.

[0069] 4. Partially self-consistent: For any non-empty directory file, a complete index table can be constructed by taking out all the rows corresponding to the non-empty directory file and all its child nodes in the index table.

[0070] In this embodiment, the index table records a wealth of file information, including file metadata, topology information, and skip list pointers. Files in the file system are arranged in "index order" in the index table. The index order follows the following rules: Parent nodes are placed before their child nodes in the index table, and parent and child node records are stored adjacent to each other. Files in the same directory are sorted in the index table in the same order as their file names. It is understood that traversing the file system's directory tree using a depth-first search, with multiple child nodes of a parent node in the directory tree sorted by file name, satisfies this index order.

[0071] Based on the index order arrangement rules, you can easily compare two file paths and determine their order in the index table by comparing the lexicographic order of the first different nodes in the two paths. For example, the paths / home / admin / apps / app1 / config and / home / admin / apps / logs have the same path prefix node / home / admin / apps, and the first different nodes are app1 and logs, respectively. Since app1 has the lexicographic order first, the file / home / admin / apps / app1 / config is ranked before / home / admin / apps / logs in the index table.

[0072] The technical solution provided by the embodiments of the present application utilizes ordered table storage, isomorphic rows, and embedded skip table pointers when creating a file system index table. This creates an index table with high storage efficiency and supports efficient sequential and random access. Furthermore, creating the file system index table requires only a single traversal of the file system, minimizing I / O (Input / Output) operations. Sequential read and write I / O is the primary focus, resulting in high creation efficiency.

[0073] In some embodiments, after the current index table is created, the current index table may be accessed in a sequential access manner or a random access manner to read the file information in the current index table.

[0074] In this embodiment, sequential access means sequentially traversing each row in the current index table to obtain the file information of each row in the current index table. As an example, in order to efficiently access the index table sequentially, the current index table is accessed in a sequential access manner, and the implementation method for reading the file information in the current index table is as follows: traversing the current index table row by row, and for the current row in the current index table traversed, reading the file information in the current row, and reducing the number of sub-files in the current top element of the context stack by 1; generating the file path of the file corresponding to the current row based on the file names recorded in all stack elements in the context stack and the file name corresponding to the current row; if the current row corresponds to a non-empty directory file, then the file name of the non-empty directory file and the number of sub-files thereof are pushed as the new top element into the context stack; if the current row corresponds to an empty directory file or a non-directory file, and the file corresponding to the current row is the last sub-file of the non-empty directory file at the previous level, then all stack elements in the context stack with a sub-file number of 0 are popped out, and the step of traversing the current index table row by row is returned to, until all rows in the current index table are traversed.

[0075] Specifically, when sequentially accessing the current index table, the metadata for each row of records is read from the current index table. Since each row only records the file name and not the file path, the file path of each file is dynamically restored by maintaining the context stack during the traversal process. During the traversal of the current index table, the file names recorded by all stack elements in the context stack and the file name corresponding to the current row are concatenated using a path separator (e.g., / ) to obtain the file path corresponding to the file in the current row.

[0076] by Figure 3For example, the context stack maintains the file name (fileName) and the number of unvisited child files (unvisitedchildren num). The current index table is traversed line by line. First, the 0th row of the current index table is traversed, that is, the file information of the home node is read from the current index table. Since the home node is the first node traversed, the context stack does not have any stack elements yet. The file name of the home node and the number of unvisited child files (6) are directly pushed into the context stack as the new top element. Next, the 1st row of the current index table is traversed, that is, the file information of the admin node is read from the current index table. Since the context stack already has a top element, the number of unvisited child files of the current top element home in the context stack needs to be reduced by 1, that is, the number of unvisited child files of the top element home is updated to 5, and home and admin are connected by a path separator (for example, / ) to obtain the file path corresponding to the current row / home / admin; and the file name of the admin node and the number of unvisited child files (1) are pushed into the context stack as the new top element. Next, traverse the second row of the current index table, that is, read the file information of the logs node from the current index table, subtract 1 from the number of unaccessed subfiles of the current top element admin in the context stack, that is, update the number of unaccessed subfiles of the top element home to 0, and connect / home / admin / logs with a path separator (for example, / ) to obtain the file path corresponding to the current row, / home / admin / logs; and push the file name and number of unaccessed subfiles (1) of the logs node into the context stack as the new top element. Finally, traverse the third row of the current index table, that is, read the file information of the 1220.log node from the current index table, subtract 1 from the number of unaccessed subfiles of the current top element logs in the context stack, that is, update the number of unaccessed subfiles of the top element logs to 0. Concatenate the directory names (home, admin, and logs) from the bottom to the top of the context stack using the path separator / to obtain the parent directory path of the 1220.log file. Concatenate the parent directory path of the 1220.log file with the 1220.log file to obtain the file path of the 1220.log file: / home / admin / logs / 1220.log. After restoring the file path of the 1220.log file, since the 1220.log file is a non-directory file and the last child file of its directory file, all consecutive stack elements starting from the top element in the context stack with a zero number of unvisited child files are popped from the context stack to update the context stack.The updated context stack has only one stack element, which is the name of the home file and the number of its unvisited subfiles (5), which means that there are still 5 subfiles of the home file that have not been accessed.

[0077] See also Figure 4 The current index table is traversed line by line. If the traversal reaches the fourth row in the current index table, that is, the file information of the ca i node is read from the current index table, the number of unaccessed subfiles of the current top element home in the context stack is reduced by 1, that is, the number of unaccessed subfiles of the top element home in the context stack is updated to 4. The home directory names from the bottom to the top of the context stack are concatenated with the ca i file name using the path separator / to obtain the file path of the ca i file / home / ca i.

[0078] After reading the file information of the ca i node from the current index table, since there are still four files in the home directory that have not been traversed, namely root, note.txt, readme.md, and sound.mp3, the number of unvisited sub-files of home in the context stack is 4. Similarly, the file paths of all files can be restored.

[0079] In this embodiment, the current index table is accessed in a random access manner, that is, any row of the current index table can be accessed.

[0080] As an example, in order to support efficient random access to the index table, the current index table is accessed in a random access manner to read the file information in the current index table as follows: query the current index table according to the row number of the file to be retrieved to obtain the file information of the file to be retrieved.

[0081] Specifically, because each row in the index table is homogeneous, the row number can be used to quickly locate the row in the index table and read the file information recorded in that row. Therefore, once the row number of the file to be retrieved in the file system (i.e., the file to be retrieved) is known, the file information of the file to be retrieved can be efficiently obtained by searching the current index table based on the row number.

[0082] As another example, in order to support efficient random access to the index table, the current index table is accessed in a random access manner to read the file information in the current index table: the first file name in the file path to be retrieved is used as the current file name, and the jump table of the current file name is obtained according to the next pointer of the subfile under the current file name recorded in the current index table; the next file name of the current file name is searched in the jump table of the current file name; the next file name is used as the new current file name, and the step of obtaining the jump table of the current file name according to the next pointer of the subfile corresponding to the current file name recorded in the index table is returned to, until the current file name is the second to last file name in the file path to be retrieved; the row number of the last file name in the file path to be retrieved is searched in the jump table of the second to last file name, and the current index table is searched according to the row number of the last file name to obtain the file information corresponding to the file path to be retrieved.

[0083] Specifically, based on the skip list, the row in the index table can be quickly located and the file information recorded in the row can be read. A skip list of the parent node can be formed based on the next-level child node of the parent node in the directory tree. Taking the home node on the directory tree as an example, the next-level child nodes of the home node, such as the admin node, cai node, note.txt node, readme.md node, root node, sound.mp3, etc., constitute the skip list of the home directory file. The skip list includes three layers of linked lists. Among them, the next3 pointer of the admin node in the third-level linked list points to the root node in the third-level linked list, and the pointer value of the next3 pointer of the root node in the third-level linked list is NUL. The next2 pointer of the admin node in the second-level linked list points to the note.txt node in the second-level linked list, and the next2 pointer of the note.txt node in the second-level linked list points to the root node in the second-level linked list, and the pointer value of the next2 pointer of the root node in the second-level linked list is NUL. The next1 pointer of the admin node in the first-level linked list points to the cai node in the first-level linked list, the next1 pointer of the cai node in the first-level linked list points to the note.txt node in the first-level linked list, the next1 pointer of the note.txt node in the first-level linked list points to the readme.md node in the first-level linked list, the next1 pointer of the readme.md node in the first-level linked list points to the root node in the first-level linked list, the pointer value of the next1 pointer of the root node in the first-level linked list points to the sound.mp3 node in the first-level linked list, and the pointer value of the next1 pointer of the sound.mp3 node in the first-level linked list is NUL.

[0084] When retrieving file information for a file path to be retrieved, if the first file name in the file path to be retrieved is the same as the file name in the first row of the index table, the file information for the file path to be retrieved can be retrieved using the skip table method. If the first file name in the file path to be retrieved is different from the file name in the first row of the index table, the file information for the file path to be retrieved cannot be retrieved using the skip table method.

[0085] Specifically, take the path of the file to be retrieved (that is, the file path of the file to be retrieved) as / home / sound.mp3. First, due to the home and Figure 2 If the file name of row number 0 in the index table is the same as the file name of the file "home", then search for sound.mp3 in the skip list of the file "home". The root node in the third-level linked list is found through the next3 pointer of the admin node in the third-level linked list in the skip list of the file "home". The lexicographic order of the root file and the sound.mp3 file is compared. The lexicographic order of the root file and the file name of the root file is higher, indicating that the sound.mp3 file to be found is after the root file. At this time, the root file is the last node in the second-level linked list, so the query is performed in the second-level linked list. The root pointer in the second-level linked list is NUL, so we continue to go to the first-level linked list. We search backward in the first-level linked list and find that the next node is the file sound.mp3 to be found. According to the row number of the file sound.mp3 recorded in the skip list, we search the index table and obtain the file information of the file sound.mp3.

[0086] The query of file paths at more levels is similar. The file path to be retrieved can be split into several layers according to the path separator (" / "), and queried layer by layer according to the embedded skip table pointer. After querying the index row of an intermediate node on the file path to be retrieved, the next row is the first node of its child node skip table. According to the order of the nodes on the file path to be retrieved, first search the skip table of each node, and then search for the next node in the skip table of the node. For example, the file path to be retrieved is: / home / admin / apps / app1 file. You need to first locate admin in the skip table of home, then locate apps in the skip table of admin, and finally locate app1 file in the skip table of apps.

[0087] In some embodiments, after the creation of the current index table is completed, the historical index table and the current index table can also be compared to obtain change information of all or part of the files in the file system. Of course, file information can also be read through the comparison.

[0088] Specifically, the historical index table is created earlier than the current index table. As time goes by, index tables corresponding to different times can be created. By comparing the historical index table with the current index table, you can obtain change information for all files in the file system, mark newly added, modified, and deleted files, and thus support file retrieval under complex conditions. For example, compared to the last scan of the file system, retrieve txt files (text files) whose file size has increased by more than 10MB (megabytes); compared to the last scan of the file system, retrieve newly added files for which the owner does not have read permission.

[0089] As an example, the implementation method of comparing the historical index table and the current index table to obtain the change information of all or part of the files in the file system is as follows: traverse the same rows in the current index table and the historical index table starting from row 1; determine the file change information based on the comparison result between the file index sequence corresponding to the same row in the current index table and the file index sequence corresponding to the same row in the historical index table; determine the file change information based on the order in which the traversal of the current index table and the historical index table is completed.

[0090] Specifically, since the index table records the file information of the files in the file system at a certain moment, by comparing the index tables of the file system at different times, the change information of the files in the file system can be obtained. Assume that the current index table is recorded as index table prev, and the historical index table is recorded as index table curr. Start traversing the two index tables from the first row. When traversing, maintain respective traversal context stacks for the two indexes respectively, and restore the file path of each file through the traversal context stack. Since the file information of the file is recorded in the index table according to the index sequence (that is, the file index sequence), compare the "index sequence" of the two files in the current row traversed in the index table prev and the index table curr. Assume that the "index sequence" of the current row traversed in the index table prev is recorded as p, and the "index sequence" of the current row traversed in the index table curr is recorded as c. If p is less than c, the file corresponding to p has been deleted from the file system; if p is greater than c, the file corresponding to c is a newly added file in the file system. If p is equal to c, it means that the file information of the same file exists in both the index table prev and the index table curr. By comparing the metadata of the file in the index table prev with the metadata of the file in the index table curr, the change information of the file is determined.

[0091] For example, see Figure 6 , the root file is the deleted file, and the 1221.log file is the newly added file. If the same row traversed corresponds to the / home / ca i file in the index table prev,

[0092] The same row found in the index table curr corresponds to the file / home / admin / logs / 1221.log. Since the index sequence of / home / ca i is greater than that of / home / admin / logs / 1221.log, the file / home / admin / logs / 1221.log is determined to be a newly added file. During the traversal process, the file path of each file is restored by traversing the context stack.

[0093] In this embodiment, if the index table prev has been traversed and the index table curr has not been traversed, the files corresponding to all the remaining rows in the index table curr are newly added files. Similarly, if the index table prev has not been traversed and the index table curr has been traversed, the files corresponding to all the remaining rows in the index table prev are deleted files. For a specific example, see Figure 7 , the index table curr has been traversed, and the / home / root / file in the index table prev has not been traversed. At this time, the / home / root / file is determined to be a deleted file.

[0094] In some embodiments, because the index table is locally self-consistent, meaning that the rows corresponding to any directory and all its subfiles can form an independent index table, the index table can be split to create sub-index tables. This local self-consistency allows for efficient subtask splitting during file detection, thereby improving task parallelism.

[0095] Exemplarily, after completing the creation of the current index table, the row number of the sub-file of the non-empty directory file is determined based on the row number of the non-empty directory file and the jump list of the non-empty directory file; based on the row number of the non-empty directory file and the row number of its sub-file, the row of the non-empty directory file and the row of the sub-file are extracted from the current index table, and a sub-index table is generated based on the row of the non-empty directory file and the row of the sub-file.

[0096] by Figure 8 For example, the admin non-empty directory files include config files, logs files, 1220.log files and other sub-files. Figure 8 Extract the rows of admin files, config files, logs files, and 1220.log files from the index table shown to form a complete sub-index table. Figure 8 The rows in red constitute the sub-index table.

[0097] In some embodiments, the index table contains topological structure data, metadata, and jump lists, etc. The index table has the characteristics of ordered storage and isomorphism of each row, which makes it easy to expand the index table to support indexing richer file information. The basic idea of ​​extending the index table is to use the original index table as the main index and establish an auxiliary index table in the form of key-value pairs, where the key is the row number, pointing to the original main index, and the value is the extended index data of the file. There can be multiple auxiliary index tables. Since the auxiliary index table does not necessarily contain records of all files, the auxiliary index table can be used for scenarios where the extended file information is not applicable to all files. For example, an auxiliary index table is established to classify files, and file types include executable files, audio, video, text, etc. Each row in the auxiliary index table stores the file type and the row number in the main index table. When searching the auxiliary index table, when a file of a specified type is queried, the target file information can be found in the main index by the row number, and the file path of the entire file can be restored by the parent pointer.

[0098] Based on the above, after completing the creation of the current index table, we also use the current index table as the main index table to create an auxiliary index table in the form of key-value pairs. The key in the auxiliary index table is the row number, and the value in the auxiliary index table is the extended index data of the file. The extended index data can be flexibly set as needed. Figure 9 The key of the auxiliary index (also known as the complex index table) is the line number (also known as the row sequence number), and the value is the extended index data (for example, the file classification). The extended index data at line 4 in the auxiliary index is the file classification of the 1220.log file at line 4 in the primary index (also known as the primary index table); the extended index data at line 6 in the auxiliary index is the file classification of the check file at line 6 in the primary index; and the extended index data at line 8 in the auxiliary index is the file classification of the soud.mp4 file at line 9 in the primary index.

[0099] In some embodiments, considering that the detection item tasks using indexes may run on different platforms and may have many differences in hardware architecture, operating system, programming language, etc., to ensure the portability of the index table, the index table can be exported as a plain text file to facilitate transmission and cross-platform loading. In addition, since the index table is logically an ordered table, it can be easily exported as a table file. Common formats are CSV (comma-separated values) format and TSV (tab-separated values) format.

[0100] In some embodiments, the index table is logically represented as an ordered table. Considering the large number of files, the entire index table cannot be loaded into memory. Therefore, the index table can be stored in pages. Therefore, after the current index table is created, it can be stored in pages on disk as an index file.

[0101] Specifically, an index file consists of a file header and several "pages," each of fixed size, called an "index page." Unlike operating system pages, which are typically 4KB (kilobytes), index pages are configurable, such as 40MB or 200MB, but must be an integer multiple of the memory page size for efficient loading.

[0102] Optionally, the index file includes an index header and multiple index pages, the index header includes an index file identifier and the total number of files; the index page includes a page header area, a file information area and a file name storage area, the page header area is used to store the file number of files recorded in the index page, the file information area is used to store file information of the files recorded in the index page, and the file name storage area is used to store the file names of the files recorded in the index page.

[0103] Specifically, the index header is the file header of the index file, and includes, but is not limited to, the index file identifier, the total number of files, and the number of files corresponding to each index page. The index file identifier can be a magic number that identifies the index file; the total number of files in the index header is the number of files recorded in the index table. Since each row in the index table corresponds to one file, the total number of files in the index header can also be the total number of rows in the index table.

[0104] The index page is divided into three parts: the page header area, the file information area and the file name storage area. The page header area is used to store the file number of the files recorded in the index page; the file information area includes several index rows of equal length, and the index rows are the rows in the index page that store the file information in the index table; the file name storage area stores the file name of the file recorded in the index page, which can be a series of strings ending with \0.

[0105] Optionally, the file name storage area is located at the end of the index page, the index row in the file information area is provided with a pointer to the corresponding file name, the index row in the file information area grows from the beginning of the page to the end of the page, and the rows in the file name storage area grow from the end of the page to the beginning of the page.

[0106] In practical applications, a simple implementation of an index page is to allocate space of the same fixed size for each row, which makes it convenient to randomly access the index row through the row number. However, since the file name is not a fixed length, it is necessary to reserve space for each row to accommodate the longest file name, which will cause a lot of space waste. Therefore, the index page adopts a solution to store the file name separately from other file information. Other file information includes metadata, topology information other than the file name, jump table, etc. All file names in the index page are stored at the end of the index page, and the index row in the file information area references the file name through a pointer to the corresponding file name. The index row grows from the top of the page to the end of the page, while the file name storage area grows in the opposite direction, allocating space from the back to the front. When the remaining space at the end is not enough to accommodate a new row and its file name, a new page space is opened up. Figure 10 Taking the index page shown as an example, each index row includes the file name field, subfile number (subfile num), parent pointer (parent), file type (type), file size (size), file owner (owner), file creation time (ctime), file last access time (atime), file last modification time (mtime), and a skip list. The file name field in each index row is a pointer to the file name in the index page. The file name pointer in the first index row points to the file name "home" in the file storage area at the end of the index page; the file name pointer in the second index row points to the file name "admin" in the file storage area at the end of the index page; and the file name pointer in the third index row points to the file name "logs" in the file storage area at the end of the index page.

[0107] One advantage of index page storage is efficient access. Each row has a fixed length, enabling quick locating of rows based on their sequence numbers. Because each page header stores the number of files recorded on that page, the page containing the row number can be quickly found by row number, and then the row within the page can be found using a modulo operation. Page storage also accommodates the variable length of file names, eliminating the need to preallocate excess space for long file names, significantly reducing space waste.

[0108] In some embodiments, one of the goals of an efficient indexing scheme is to support indexing a very large number of files. Considering that in scenarios where the number of files is extremely large, the index table cannot be fully loaded into the memory, paging storage is used to load the pages into the memory. Optionally, after the current index table is stored in the form of an index file in the form of pages on the disk, the index header is also loaded from the disk into the memory, and the index pages are loaded from the disk into the index page pool in the memory in a paging loading manner, and the index pages in the index page pool are maintained and managed using a cache eviction algorithm.

[0109] Specifically, in the index table, parent nodes and their child nodes are stored adjacent to each other, and all child nodes under the same parent node are stored adjacent to each other. This feature makes access to the index table have good spatial locality. In other words, when a row in the index table is read, its adjacent rows are likely to be read immediately. The index file is loaded page by page, so the index row to be accessed may already exist in the memory. Only some index pages can be loaded into the memory, and cache eviction algorithms such as LRU (Least Recently Used) can be used to maintain some of the most frequently accessed index pages in the memory, thereby improving the efficiency of index access.

[0110] Optionally, to further accelerate index reading and writing, after loading index pages from disk into the index page pool in memory using paging loading, the row sequence numbers and row data of the most frequently accessed index rows are cached in the row cache table in memory, and the row cache table is maintained and managed using a cache eviction algorithm.

[0111] Specifically, to further accelerate index reads and writes, the efficient indexing solution also introduces a row cache table. This cache stores the row numbers and row data of the most frequently accessed index rows as key-value pairs. The row cache table is also managed using a cache eviction algorithm to ensure the freshness of the cached data.

[0112] See also Figure 11 The index file stored on disk consists of an index header and multiple index pages. The index file is loaded from disk into memory. The row cache table and index page pool form a two-level cache. The row cache table key is the row number (Rowindex), and the row cache table value is the row data (Row). The index header resides in memory and is used to quickly locate the index page based on the row number. When searching for a row in an index page, the row cache table is first accessed. If the row does not exist there, the index pool is accessed and the row cache table is updated. If the row does not exist in the index page pool, the index page is loaded from the index file into memory. When the index page pool reaches its capacity limit, loading a new index page causes some existing index pages to be evicted, and these evicted index pages are written back to disk. When writing an index, only the row cache table is written. When the cache is evicted, the data is written back to the index page pool, one level down. That is, when a row record is evicted from the row cache table, it is written back to the index page pool. When an index page is evicted from the index page pool, it is written back to the disk file. This read and write solution ensures high read and write efficiency because in most cases, only the row cache table needs to be read and written.

[0113] The efficient file system indexing scheme provided by the embodiment of the present application has the advantages of high retrieval efficiency, low resource usage, portability, and easy splitting. In terms of index logic, ordered table storage is used, each row is isomorphic, and a skip table pointer is embedded, so the sequential traversal and random access efficiency are high. This indexing scheme supports indexing rich file information and supports expansion and historical index result comparison, so it can support complex conditional retrieval. In terms of physical storage, paging storage and paging loading schemes are used to reduce space waste. In terms of storage structure, the storage form of the index table is efficient, and the principle of memory access locality is fully utilized for optimization, which reduces space waste; efficient access is achieved through dynamic segmented loading, and a large number of files can be indexed at one time with limited memory resource overhead. This indexing scheme can meet the requirements of high query performance, support for complex retrieval, and portability in various file detection scenarios, and can effectively support the implementation of high-performance file detection.

[0114] Figure 12 This is a flowchart of a file retrieval method based on a file system index provided by an embodiment of the present application. Figure 12 , the method may include the following steps:

[0115] 201. Access the current index table of the file system to read file information of each file in the current index table.

[0116] For details on how to create and access the current index table, please refer to the relevant content of the aforementioned embodiment.

[0117] 202. Filter the file information of each read file according to the search condition to obtain target file information that meets the search condition.

[0118] 203. Retrieve the target file corresponding to the target file information in the file system.

[0119] Further optionally, filtering the file information of each read file according to the search conditions includes: comparing the historical index table and the current index table to obtain change information of all files in the file system; labeling the file information of each read file, the label reflecting the change information of the file; and filtering the file information of each tagged file according to the search conditions.

[0120] It is worth noting that the file information of the file is tagged through index table comparison, and the files whose tags meet the search conditions are retrieved from the file system during the search process, thereby supporting more complex conditional searches.

[0121] In this embodiment, the comparison method of the index table can refer to the relevant content of the above embodiment.

[0122] Specifically, during the file retrieval stage, one index table (also known as the current index table) or two index tables (also known as the current index table and the historical index table) can be used to dynamically obtain file information of multiple files, and then, based on the input retrieval conditions, the file information that meets the retrieval conditions is filtered out from the file information of multiple files, and the files corresponding to the file information that meets the retrieval conditions are retrieved in the file system. Optionally, considering that in a scenario with a huge number of files, the number of files that meet a certain retrieval condition may be huge and cannot be cached in memory, file retrieval supports streaming reading of results, that is, returning retrieval results in a lazy loading manner, and returning the next retrieval result only after the inquirer reads a retrieval result.

[0123] For ease of understanding, combined Figure 13 Describes an example file retrieval process. Figure 13 , from the current index table ( Figure 13 Read the file information of each file in the file system from the index prev in the file system, and compare the current index table with the historical index table ( Figure 3 The index curr in the file system is used to obtain the change information of each file in the file system; the file information of each file in the read file system is marked with a label representing the change information, for example, Figure 13 The file whose file path (also called full path) is / home / admin has the following change information: unmodified. Figure 13 The change information of the file with the file path / home / admin / logs / 1220.log is newly added. Figure 13 In the example, the file information stream includes multiple file information, and the file information in the file information stream is subjected to a composite condition judgment according to the search condition, that is, the file information that meets the search condition is filtered out from the file information in the file information stream (that is, the file information that meets the search condition is filtered out). Figure 13 (File information for compound conditions in ).

[0124] The technical solution provided by the embodiments of this application provides a rich file information index table in the file system, which can support file retrieval under complex and compound conditions, and has high file retrieval efficiency. It can achieve "one-time traversal of the file system and search at any time" during file detection, greatly reducing the number of I / O operations during detection and shortening detection time.

[0125] In some embodiments, security checks are also supported for target files retrieved from the file system. In file detection scenarios, full utilization of index table features such as on-demand loading, ordered storage, local self-consistency, and convenient segmentation can improve file detection throughput and shorten detection time under limited resource usage, thereby providing a high-performance file detection solution.

[0126] Specifically, the efficient file system indexing solution provided by the embodiments of the present application can support various file detection scenarios. In some file detection scenarios, a file system indexing solution that meets the following conditions is required:

[0127] 1. Supports file retrieval with complex conditions. Some examples of complex condition retrieval include:

[0128] 1.1 Compared with last month, the file size increased by more than 10M and the files ending with doc.

[0129] 1.2 Files in the specified directory that have been modified in the last 24 hours and whose owners have executable permissions.

[0130] 1.3 A directory with a path depth of 4, a parent directory node that meets a specific regular expression, and contains more than 100,000 files.

[0131] 1.4 Query the total number of files in a directory.

[0132] 2. Efficient use of memory and disk storage. It can complete the indexing of a very large number of files (hundreds of millions) within limited memory, and the resulting index file cannot be too large (less than GB), to facilitate the distribution of index files.

[0133] 3. High performance. It can complete index building and conditional file retrieval in a short time.

[0134] 4. Support multi-process concurrent reading / query.

[0135] 5. Extensible. In addition to indexing basic file metadata, it can also be extended to support indexing more file information, such as file type.

[0136] In file detection, a single detection typically includes multiple detection tasks, each of which is responsible for detecting a specific type of security risk. Some detection tasks require searching for files by path, and the file information can be quickly located through the embedded skip table pointer in the index table. Some detection tasks first query the auxiliary index to obtain the line number, and then obtain the file information from the main index based on the line number. In detection scenarios such as those that require traversing the entire file system, detection performance can be improved through horizontal expansion in the form of streaming retrieval design and sub-task splitting. The streaming design improves the system's throughput, reduces system resource consumption, and avoids managing large amounts of file information in memory. Horizontal expansion enables the system to support the detection of extremely large file systems. When the number of files is so large that the performance of a single machine cannot support it, the file information can be distributed to multiple hosts in the form of index files, detection tasks can be run on multiple hosts, file retrieval can be performed on different hosts, and then files can be obtained and accessed on demand, achieving system scalability at a lower cost.

[0137] During the file detection phase, detection task instances can be abstracted into independent processes. The detection task retrieves file information that meets the search criteria through indexing and consumes this information in the form of a "stream," obtaining target files that meet the criteria from the file system for security detection. Each detection task can have multiple subtasks, or subprocesses, which are split using the Fork-Join model. The Fork-Join model refers to a parallel execution mode that splits a large task into multiple subtasks, executes them in parallel, and merges the results of the multiple subtasks to obtain the task result of the large task.

[0138] The local self-consistency of the index table facilitates subtask splitting. Simply obtaining the number of subfiles in a directory file allows determining whether a subtask needs to be assigned to the current directory file. For example, a resource pool can be maintained, containing resources that can be assigned to tasks. Upon completion of each task, the resources are returned to the resource pool. To this end, a file detection method is provided. Specifically, when executing a file retrieval task, the current index table of the file system is traversed from top to bottom. If the file information of a directory file is currently traversed, and the number of subfiles in the directory file exceeds a set threshold, a determination is made as to whether the current remaining resources in the resource pool meet the resource requirements for executing the subtask. If so, a sub-index table is generated from the current index table. The sub-index table contains the file information of the directory file and its subfiles. A subtask is created based on the sub-index table, and resources from the resource pool are allocated to the subtask to obtain a detection result for the subtask. The subtask is then executed based on the allocated resources. After the subtask is completed, the allocated resources are returned to the resource pool. The detection results of each subtask are merged to obtain a final detection result.

[0139] A resource pool can be understood as a collection of available hardware resources, including but not limited to computing resources, storage resources, and network resources. The current remaining resources in a resource pool refer to the amount of resources in the resource pool that have not yet been allocated to a subtask, while the resource demand refers to the amount of resources required to execute a subtask.

[0140] When executing a subtask based on the allocated resources, corresponding files may be retrieved from the file system based on the sub-index table corresponding to the subtask, and the retrieved files may be tested to obtain a test result.

[0141] The following combination Figure 14 This section introduces the file detection scenario implementation. Figure 14 , when checking files, a single traversal of the file system's directory tree (i.e. Figure 14), builds an index based on the result of a single traversal, that is, builds an index table of the file system. File detection may include multiple detection tasks such as detection task 1 and detection task 2. Subtasks are split based on the local self-consistency of the index table. For example, detection task 1 is split into n subtasks such as subtasks 1 to n, where n is a positive integer, and detection task 2 is split into n subtasks such as subtasks 1 to n. Based on the sub-index table corresponding to the subtask, the corresponding file is retrieved from the file system, and the retrieved file is detected to obtain the detection result, which is reported, and the reported detection results of each subtask are merged to obtain the detection result of the detection task. By using an efficient indexing scheme, "one traversal and retrieval at any time" can be achieved during file detection, which greatly reduces the number of I / O times during detection and shortens the detection time. By using the local self-consistency of the index table to split the subtasks and adopting a streaming retrieval design, the throughput and performance of the detection can be improved. In addition, the file retrieval system can be separated from the file system itself. With the portability of the index, the detection task can be distributed to different hosts. The detection task reads files on demand according to the retrieval results, which increases the scalability of the detection task.

[0142] Figure 15 This is a schematic diagram of the structure of an index processing device for a file system provided in an embodiment of the present application. Figure 15 , the apparatus may include:

[0143] A traversal module 51 is configured to traverse the directory tree of the file system using a depth-first search method to obtain metadata and topology information of multiple files in the file system;

[0144] A recording module 52, configured to record file information of multiple files in the current index table in sequence according to the traversal order of the multiple files, the file information including metadata and topological structure information of the files;

[0145] An acquisition module 53 is configured to obtain a jump list of a non-empty directory file according to the file information of the subfiles of the non-empty directory file in the directory tree;

[0146] The adding module 54 is used to add the next pointer of the sub-file in the jump list of the non-empty directory file to the row where the sub-file is located in the current index table to complete the creation of the current index table, and the next pointer points to the next node of the same-level linked list.

[0147] Further optionally, the above-mentioned device also includes: an access module, which is used to access the current index table in a sequential access manner or a random access manner to read the file information in the current index table.

[0148] Further optionally, when the access module accesses sequentially, it is specifically used to: traverse the current index table line by line, and for the current line in the current index table traversed, read the file information in the current line, and reduce the number of sub-files in the current top element of the context stack by 1; generate the file path of the file corresponding to the current line according to the file names recorded by all stack elements in the context stack and the file name corresponding to the current line; if the current line corresponds to a non-empty directory file, the file name of the non-empty directory file and the number of sub-files thereof are pushed into the context stack as the new top element; if the current line corresponds to an empty directory file or a non-directory file, and the file corresponding to the current line is the last sub-file of the non-empty directory file of the previous level, all stack elements with a sub-file number of 0 in the context stack are popped out, and return to the step of traversing the current index table line by line until all lines in the current index table are traversed.

[0149] Further optionally, when the access module performs random access, it is specifically used to: query the current index table according to the row number of the file to be retrieved to obtain the file information of the file to be retrieved.

[0150] Further optionally, during random access, the access module is specifically configured to: use the first file name in the path of the file to be retrieved as the current file name, obtain a jump list of the current file name based on the next pointer of the subfile under the current file name recorded in the current index table; search the jump list of the current file name for the next file name after the current file name; use the next file name as the new current file name, and return to the step of obtaining the jump list of the current file name based on the next pointer of the subfile corresponding to the current file name recorded in the index table, until the current file name is the second-to-last file name in the path of the file to be retrieved; search the jump list of the second-to-last file name for the row number of the last file name in the path of the file to be retrieved, and search the current index table based on the row number of the last file name to obtain file information corresponding to the path of the file to be retrieved.

[0151] Further optionally, the above device further includes: a comparison module, used to compare the historical index table and the current index table to obtain change information of all or part of the files in the file system.

[0152] Further optionally, a comparison module is specifically used to: traverse the same rows in the current index table and the historical index table starting from row 1; determine the file change information based on the comparison result between the file index sequence corresponding to the same row in the current index table and the file index sequence corresponding to the same row in the historical index table; determine the file change information based on the order in which the traversal of the current index table and the historical index table is completed.

[0153] Further optionally, the above-mentioned device also includes: a splitting module, which is used to determine the row number of the sub-file of the non-empty directory file based on the row number of the non-empty directory file and the jump list of the non-empty directory file; according to the row number of the non-empty directory file and the row number of its sub-file, extract the row of the non-empty directory file and the row of the sub-file from the current index table, and generate a sub-index table based on the row of the non-empty directory file and the row of the sub-file.

[0154] Further optionally, the above device also includes: an index extension module, which is used to use the current index table as the main index table to establish an auxiliary index table in the form of key-value pairs, where the key in the auxiliary index table is the row number and the value in the auxiliary index table is the extended index data of the file.

[0155] Further optionally, the above-mentioned device also includes: a storage module, which is used to store the current index table in the form of an index file in pages to the disk; wherein, the index file includes an index header and multiple index pages, and the index header includes an index file identifier and a total number of files; the index page includes a page header area, a file information area and a file name storage area, the page header area is used to store the file number of the files recorded in the index page, the file information area is used to store the file information of the files recorded in the index page, and the file name storage area is used to store the file names of the files recorded in the index page.

[0156] Further optionally, the file name storage area is located at the end of the index page, the index row in the file information area is provided with a pointer to the corresponding file name, the index row in the file information area grows from the beginning of the page to the end of the page, and the rows in the file name storage area grow from the end of the page to the beginning of the page.

[0157] Further optionally, the above-mentioned device also includes: a loading module, which is used to load the index header from the disk into the memory, and load the index page from the disk into the index page pool of the memory in a page loading manner, and use a cache elimination algorithm to maintain and manage the index pages in the index page pool.

[0158] Further optionally, the loading module is further used to cache the row sequence number and row data of the most frequently accessed index row in the row cache table in the memory, and to maintain and manage the row cache table using a cache elimination algorithm.

[0159] Figure 15 The device shown can perform Figure 1 The method shown in the embodiment shown, its implementation principle and technical effect are not described in detail. Figure 15 The specific manner in which each module and unit performs operations in the device shown has been described in detail in the embodiment of the method and will not be elaborated here.

[0160] Figure 16 This is a structural diagram of a file processing device provided in an embodiment of the present application. Figure 16 , the apparatus may include:

[0161] Access module 21 is used to access the current index table of the file system to read the file information of each file in the current index table. For the method of creating the current index table, please refer to the relevant content of the above embodiment;

[0162] The screening module 22 is used to screen the file information of each file read according to the search conditions to obtain the target file information that meets the search conditions;

[0163] The retrieval module 23 is used to retrieve the target file corresponding to the target file information in the file system.

[0164] Further optionally, the filtering module 22 is specifically used to: compare the historical index table and the current index table to obtain the change information of all files in the file system; label the file information of each read file, the label reflects the change information of the file; and filter the file information of each tagged file according to the search conditions.

[0165] Figure 16 The device shown can perform Figure 12 The method shown in the embodiment shown, its implementation principle and technical effect are not described in detail. Figure 16 The specific manner in which each module and unit performs operations in the device shown has been described in detail in the embodiment of the method and will not be elaborated here.

[0166] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 101 to 104 can be device A; for another example, the execution entity of steps 101 and 102 can be device A, and the execution entity of steps 103 and 104 can be device B; and so on.

[0167] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 201, 202, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0169] Figure 17 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 17 As shown, the computer device includes: a memory 31 and a processor 32;

[0170] The memory 31 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0171] The memory 31 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0172] The processor 32 is coupled to the memory 31 and is used to execute the computer program in the memory 31 to: execute the steps in the file system index processing method, execute the steps in the file processing method or execute the steps in the file detection method.

[0173] Further optional, such as Figure 17 As shown, the computer device also includes: a communication component 33, a display 34, a power component 35, an audio component 36 and other components. Figure 17 Only some components are shown schematically, which does not mean that the computer equipment only includes Figure 17 In addition, Figure 17The components in the dotted box are optional components, not mandatory components, and the specific components may depend on the product form of the computer device. The computer device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT (Internet of Things) device, or a server device such as a conventional server, a cloud server or a server array. If the computer device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it may include Figure 17 If the computer device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include the components in the dotted box; Figure 17 Components within the dotted box.

[0174] The detailed implementation process of the processor executing each action can be found in the relevant description in the aforementioned method embodiment or device embodiment, and will not be repeated here.

[0175] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by a computer device in the above method embodiment.

[0176] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement the steps that can be executed by a computer device in the above method embodiment.

[0177] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2 Generation, 2 Generation), 3G (3 Generation, 3 Generation), 4G (4 Generation, 4 Generation) / LTE (Long Term Evolution), 5G (5 Generation, 5 Generation) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0178] The above-mentioned display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0179] The power supply assembly provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0180] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0181] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0182] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0185] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0186] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0187] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Phase Change RAM (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0188] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0189] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A file system index processing method, characterized in that: include: Using a depth-first search method, traverse the directory tree of the file system to obtain metadata and topological structure information of multiple files in the file system; Recording file information of the plurality of files in the current index table in sequence according to the traversal order of the plurality of files, the file information including metadata and topological structure information of the files; Obtaining a jump list of a non-empty directory file according to file information of a subfile of the non-empty directory file in the directory tree; The next pointer of the sub-file in the jump list of the non-empty directory file is added to the row of the sub-file in the current index table to complete the creation of the current index table, and the next pointer points to the next node of the same-layer linked list.

2. The method according to claim 1, characterized in that After completing the creation of the current index table, including: The current index table is accessed in a sequential access manner or a random access manner to read the file information in the current index table.

3. The method according to claim 2, characterized in that Accessing the current index table in a sequential access manner to read file information in the current index table includes: Traversing the current index table line by line, and for the current line in the current index table, reading the file information in the current line, and reducing the number of sub-files in the current top element of the context stack by 1; Generate a file path of the file corresponding to the current row according to the file names recorded by all stack elements in the context stack and the file name corresponding to the current row; If the current line corresponds to a non-empty directory file, the file name of the non-empty directory file and the number of its sub-files are pushed into the context stack as a new top element; If the current row corresponds to an empty directory file or a non-directory file, and the file corresponding to the current row is the last sub-file of the non-empty directory file at the previous level, all stack elements with a sub-file number of 0 in the context stack are popped out, and the step of traversing the current index table row by row is returned to until all rows in the current index table are traversed.

4. The method according to claim 2, characterized in that Accessing the current index table in a random access manner to read file information in the current index table includes: The current index table is queried according to the row sequence number of the file to be retrieved to obtain the file information of the file to be retrieved.

5. The method according to claim 2, characterized in that Accessing the current index table in a random access manner to read file information in the current index table includes: The first file name in the path of the file to be retrieved is used as the current file name, and the jump list of the current file name is obtained according to the next pointer of the sub-file under the current file name recorded in the current index table; Searching for the next file name of the current file name in the skip list of the current file name; using the next file name as the new current file name, returning to the step of obtaining the skip list of the current file name according to the next pointer of the subfile corresponding to the current file name recorded in the index table, until the current file name is the second to last file name in the path of the file to be retrieved; The row number of the last file name in the file path to be retrieved is searched in the jump table of the second to last file name, and the current index table is searched according to the row number of the last file name to obtain the file information corresponding to the file path to be retrieved.

6. The method according to claim 1, characterized in that After completing the creation of the current index table, including: Compare the historical index table with the current index table to obtain change information of all or part of the files in the file system.

7. The method according to claim 6, characterized in that Comparing the historical index table with the current index table to obtain change information of all or part of the files in the file system, including: Starting from row 1, traverse the same rows in the current index table and the historical index table; Determining file change information based on a comparison result between a file index sequence corresponding to the same row in the current index table and a file index sequence corresponding to the same row in the history index table; The file change information is determined according to the order in which the current index table and the historical index table are traversed.

8. The method according to claim 1, characterized in that After completing the creation of the current index table, it also includes: Determining the row number of the subfile of the non-empty directory file according to the row number of the non-empty directory file and the skip list of the non-empty directory file; According to the row sequence number of the non-empty directory file and the row sequence number of its sub-file, the row where the non-empty directory file is located and the row of the sub-file are extracted from the current index table, and a sub-index table is generated based on the row where the non-empty directory file is located and the row of the sub-file.

9. The method according to claim 1, characterized in that After completing the creation of the current index table, it also includes: The current index table is used as the main index table, and an auxiliary index table in the form of key-value pairs is established. The key in the auxiliary index table is the row number, and the value in the auxiliary index table is the extended index data of the file.

10. The method according to claim 1, characterized in that After completing the creation of the current index table, it also includes: The current index table is stored in a disk in the form of an index file in pages; The index file includes an index header and multiple index pages, and the index header includes an index file identifier and a total number of files; The index page includes a page header area, a file information area and a file name storage area. The page header area is used to store the file number of the files recorded in the index page, the file information area is used to store the file information of the files recorded in the index page, and the file name storage area is used to store the file name of the file recorded in the index page.

11. The method according to claim 10, characterized in that The file name storage area is located at the end of the index page, the index row in the file information area is provided with a pointer pointing to the corresponding file name, the index row in the file information area grows from the beginning of the page to the end of the page, and the rows in the file name storage area grow from the end of the page to the beginning of the page.

12. The method according to claim 10, characterized in that After paging and storing the current index table in the form of index files on disk, the method further includes: The index header is loaded from the disk into the memory, and the index page is loaded from the disk into the index page pool of the memory in a page loading manner, and the index page in the index page pool is maintained and managed using a cache elimination algorithm.

13. The method according to claim 10, characterized in that After loading the index page from the disk into the index page pool in the memory in a page loading manner, the method further includes: The row sequence numbers and row data of the most frequently accessed index rows are cached in the row cache table of the memory, and the row cache table is maintained and managed by using a cache elimination algorithm.

14. A file processing method, characterized in that: include: Accessing a current index table of the file system to read file information of each file in the current index table, wherein the current index table is created according to the method of claim 1; Filter the file information of each file read according to the search conditions to obtain the target file information that meets the search conditions; The target file corresponding to the target file information is retrieved in the file system.

15. The method according to claim 14, characterized in that Filtering the file information of each file read according to the search conditions includes: Comparing the historical index table with the current index table to obtain change information of all files in the file system; Tagging the file information of each read file, wherein the tag reflects the change information of the file; The file information of each tagged file is filtered according to the search conditions.

16. The method according to claim 14, characterized in that Also includes: Perform a security check on the target file.

17. A file detection method, characterized in that: include: Traversing a current index table of the file system from top to bottom, wherein the current index table is created according to the method of claim 1; If the file information of the directory file is currently traversed, then if the number of sub-files in the directory file is greater than the set threshold, determine whether the current remaining resources in the resource pool meet the resource requirements for subtask execution; If the conditions are met, a sub-index table is split from the current index table, where the sub-index table includes file information of the directory file and its sub-files; Creating a subtask based on the sub-index table, allocating resources in the resource pool to the subtask, executing the subtask based on the allocated resources to obtain a detection result of the subtask, and returning the allocated resources to the resource pool after the subtask is executed; The detection results of each subtask are combined to obtain the final detection result.

18. A computer device, characterized in that: include: memory and processor; The memory is used to store computer programs; The processor is coupled to the memory and configured to execute the computer program to perform the steps of the method according to any one of claims 1 to 17.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps of the method according to any one of claims 1 to 17.

20. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, causes the processor to implement the steps of the method according to any one of claims 1 to 17.

Images