File scanning method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a file scanning method and device, electronic equipment and a computer readable storage medium, wherein the method comprises: obtaining current directory information of a target server through a first-stage scanning process; comparing the current directory information with historical directory information to determine update information in the current directory information, wherein the historical directory information is the latest version of the directory information of the target server currently stored locally; transmitting the update information to a second-stage scanning process; and processing the update information through the second-stage scanning process to update the historical directory information to obtain updated historical directory information.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a document scanning method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In big data scenarios, it is necessary to synchronize large amounts of file data across network file storage servers, enabling devices to scan, retrieve, and push files between two networks. As server disk arrays become larger and the data on storage servers grows larger, traditional data scanning methods are insufficient to support scenarios with large data volumes. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a document scanning method, apparatus, electronic device and computer-readable storage medium to improve the problem that existing traditional data scanning methods are insufficient to support scenarios with large data volumes.

[0004] In a first aspect, embodiments of this application provide a file scanning method, comprising: obtaining current directory information of a target server through a first-stage scanning process; comparing the current directory information with historical directory information to determine updated information in the current directory information, wherein the historical directory information is the latest version of the target server's directory information currently stored locally; transmitting the updated information to a second-stage scanning process; and processing the updated information through the second-stage scanning process to update the historical directory information to obtain updated historical directory information.

[0005] In the above implementation, a two-stage scanning process can be used to perform different scanning and data processing tasks, which can distribute the pressure of data processing, process multiple scanning stages separately, improve the efficiency of data scanning, and reduce the difficulty of scanning large amounts of data.

[0006] In an optional implementation, the current directory information includes: multiple file information items, and the update information includes: file update information; comparing the current directory information with historical directory information to determine the update information in the current directory information includes: comparing each of the file information items with the content recorded in the historical directory information to determine the file update information.

[0007] In the above implementation, by determining the file update information, the subsequent processing tasks caused by the file update can be handled through the second-stage scanning process, reducing the processing difficulty of the first-stage scanning process, and also allowing the tasks to be better distributed, thereby improving the data scanning synchronization efficiency.

[0008] In an optional implementation, the file update information includes: file addition information; the step of comparing each piece of file information with the content recorded in the historical directory information to determine the file update information includes: comparing each piece of file information with the content recorded in the historical directory information to determine whether the file corresponding to each piece of file information exists in the historical directory information; if the file corresponding to the first piece of file information does not exist in the historical directory information, then the file addition information is determined based on the first piece of file information, wherein the first piece of file information is any one of the multiple pieces of file information contained in the current directory information.

[0009] In an optional implementation, the file update information includes: file change information; the step of comparing each piece of file information with the content recorded in the historical directory information to determine the file update information includes: comparing each piece of file information with the content recorded in the historical directory information to determine whether the file corresponding to each piece of file information exists in the historical directory information; if the file corresponding to the second piece of file information exists in the historical directory information, and the second piece of file information is different from the file information corresponding to the historical directory information, then the file change information is determined based on the second piece of file information, wherein the second piece of file information is any one of the multiple pieces of file information included in the current directory information.

[0010] In the above implementation, different file updates can be identified separately, so as to facilitate different processing for different file updates, thereby making data scanning and processing more accurate and orderly.

[0011] In an optional implementation, the historical directory information may include: a file tree, the file tree including multiple nodes, each node representing a directory or a file; the step of comparing each piece of file information with the content recorded in the historical directory information to determine file update information includes: comparing the filenames in each piece of file information with each node of the file tree to determine whether the file corresponding to each piece of file information belongs to any node in the file tree; if the file corresponding to the third piece of file information does not belong to any node in the file tree, determining file addition information based on the third piece of file information, wherein the third piece of file information is any one of the multiple pieces of file information included in the current directory information; if the file corresponding to the third piece of file information belongs to a node in the file tree, and the third piece of file information is different from the file information to which the corresponding node in the file tree belongs, then determining file change information based on the third piece of file information.

[0012] In the above implementation method, historical directory information can be presented in the form of a file tree. The file update status can be determined by comparing nodes, which can achieve file comparison relatively quickly and intuitively.

[0013] In an optional implementation, if the file corresponding to any third file information does not belong to a node in the file tree, the method further includes: adding a new file node corresponding to the third file information in the file tree, and assigning the third file information to the new file node.

[0014] In the above implementation, when a new file is confirmed, the nodes of the file tree can be updated first. This allows the second-stage scanning process to only need to process the nodes in more detail, reducing the number of actions required in the second-stage scanning process and improving scanning efficiency.

[0015] In an optional implementation, obtaining the current directory information of the target server includes: determining a first number of scanning instances based on the number of nodes in the directory tree of the target server, wherein each scanning instance is used to scan at least one node in the directory tree, and the first number of scanning instances are used to scan the nodes contained in the directory tree; for the i-th scanning instance, obtaining the i-th directory information corresponding to the i-th part of the nodes from the target server, wherein the i-th scanning instance is any one of the first number of scanning instances, i is a positive integer greater than or equal to 1 and less than or equal to N, and N is the first number, wherein the union of the directory information obtained based on each scanning instance is the current directory information of the target server; comparing the current directory information with historical directory information to determine the updated information in the current directory information includes: comparing the i-th directory information with historical directory information through the i-th scanning instance to determine the i-th updated information in the i-th directory information, wherein the updated information in the current directory information can be obtained by comparing the current directory information with historical directory information based on each scanning instance.

[0016] In the above implementation, when the target server's directory tree has a large number of nodes, the directory tree can be divided into multiple parts, which can be processed in parallel by different scanning instances to improve the efficiency of directory scanning.

[0017] In an optional implementation, obtaining the current directory information of the target server includes: obtaining initial directory information from the target server, the initial directory information including multiple initial fields, each initial field representing the content of a file or the content of a directory; extracting the attribute fields contained in each initial field, and recording each attribute field as target format information to obtain the current directory information of the target server.

[0018] In an optional implementation, the attribute field includes a time field; the step of extracting the attribute fields contained in each of the initial fields and recording each attribute field as target format information includes: extracting the attribute fields contained in each of the initial fields; if the time field in the attribute field of the first initial field is incomplete, then correcting the time field of the first initial field to obtain a corrected time, wherein the first initial field is any one of multiple initial fields; and recording the corrected time and the attribute field of the first initial field as target format information.

[0019] The above implementation method can also identify the integrity of the extracted fields. If there are any missing fields, they can be modified, thereby better maintaining the integrity of each piece of information in the directory information and improving the accuracy of the file scanning results.

[0020] In an optional implementation, before obtaining the initial directory information from the target server, the method further includes: obtaining the signaling types supported by the target server; if the signaling types supported by the target server include at least two, then determining the target signaling according to the priority of the at least two signaling types, wherein the target signaling is used to perform the steps of obtaining the initial directory information from the target server and extracting the attribute fields contained in each of the initial fields.

[0021] In the above implementation, the server's needs can be adapted to select appropriate signaling to process the directory, thereby making the file scanning method more widely applicable.

[0022] In an optional implementation, the current directory information includes: multiple subdirectory information, and the update information includes: directory update information; comparing the current directory information with historical directory information to determine the update information in the current directory information includes: comparing each of the subdirectory information with the content recorded in the historical directory information to determine the directory update information; the method further includes: updating the directories in the historical directory information according to the directory update information.

[0023] In the above implementation, when it is determined that there is a directory update, the directory in the historical directory information can be updated directly, which can also reduce the amount of processing required in the second stage of the scanning process.

[0024] In an optional implementation, the update information includes file update information; processing the update information to update the historical directory information includes: determining a set of files to be processed based on the received file update information; determining the directories to which each file belongs in the set of files to be processed belongs, thereby defining the directories to which each file belongs as a set of directories to be processed, and determining the number of files to be processed contained in each directory in the set of directories to be processed; for a first directory to be processed, determining a file information acquisition method based on the number of files to be processed contained in the first directory to be processed, wherein the first directory to be processed is any one of the directories to be processed in the set of directories to be processed; if the number of files to be processed is greater than a specified number, using a batch acquisition method to acquire the file information of the files to be processed; if the number of files to be processed is not greater than the specified number, using a single acquisition method to acquire the file information of the files to be processed; sequentially acquiring the file information of each file to be processed from the target server according to the file acquisition method; and updating the historical directory information based on the file information of each file to be processed.

[0025] In the above implementation, different file acquisition methods are used for different data volumes, which can better balance the efficiency and accuracy of file acquisition.

[0026] In an optional implementation, the first-stage scanning process is implemented through a first process; the second-stage scanning process is implemented through a second process.

[0027] In the above implementation, two relatively independent processes can be used to handle the first-stage scanning process and the second-stage scanning process, thereby improving the efficiency of file scanning and synchronization.

[0028] In an optional implementation, obtaining the current directory information of the target server includes: obtaining the current directory information from the target server according to a preset filtering strategy, wherein the filtering strategy includes one or more of the following: directory filtering strategy, file filtering strategy, and file priority strategy.

[0029] In the above implementation method, the file scanning range can also be set according to needs to adapt to different file synchronization requirements in different scenarios.

[0030] In an optional implementation, before obtaining the current directory information of the target server, the method further includes: obtaining the target type of the file sharing protocol supported by the target server, wherein the file sharing protocol includes at least one of the following: NFS protocol, SMB protocol, FTP protocol, and SFTP protocol; and determining a scanning plugin corresponding to the target type based on the target type, wherein the scanning plugin is used to execute the first stage scanning process and the second stage scanning process.

[0031] In the above implementation, considering that different servers support different protocols, multiple scanning plugins can be preset to meet the needs of different servers.

[0032] In an optional implementation, before obtaining the current directory information of the target server, the method further includes at least one of the following: testing the connectivity with the target server; testing the permissions of the target server, wherein the permissions include one or more of the following: directory creation, directory deletion, file creation, file deletion, file read operation, and file write operation; testing the scalability of the target server, wherein the scalability includes: whether it supports modifying file attributes; if any test fails, outputting the reason for the test failure.

[0033] In the above implementation, a connection or permission test can be performed with the server before file scanning to improve the success rate of subsequent file scanning and enable users to more accurately understand the current status of the server.

[0034] Secondly, embodiments of this application provide a file scanning device, comprising: a first acquisition module, configured to acquire current directory information of a target server through a first-stage scanning process; a comparison module, configured to compare the current directory information with historical directory information to determine updated information in the current directory information, wherein the historical directory information is the latest version of the target server's directory information currently stored locally; a transmission module, configured to transmit the updated information to a second-stage scanning process; and a first update module, configured to process the updated information through the second-stage scanning process to update the historical directory information to obtain updated historical directory information.

[0035] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the above-described method.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the method described above.

[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A schematic diagram of the operating environment for the document scanning method provided in this application embodiment;

[0040] Figure 2 A block diagram illustrating an electronic device provided in an embodiment of this application;

[0041] Figure 3 A flowchart illustrating the document scanning method provided in this application embodiment;

[0042] Figure 4 A schematic diagram of the directory tree of the target server in an example is shown;

[0043] Figure 5 This is a schematic diagram of the functional modules of the document scanning device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0045] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0046] In big data scenarios, it's necessary to synchronize large amounts of file data across network file storage servers, enabling scanning, fetching, and pushing files between two networks. However, with the increasing size of disk arrays on network file storage servers and the widespread adoption of 10 Gigabit networks, traditional single-threaded scanning solutions for scanning, fetching, and pushing files across networks are insufficient to support these high-volume data scenarios.

[0047] Based on this situation, this application provides a file scanning method, apparatus, electronic device, and computer-readable storage medium that can scan remote storage servers with a multi-connection, multi-threaded structure, thereby discovering file changes in large-scale directory trees and improving scanning efficiency.

[0048] To facilitate understanding of this embodiment, the operating environment for implementing the file scanning method disclosed in this application embodiment will first be introduced.

[0049] like Figure 1 The diagram shown illustrates the operating environment of the file scanning method provided in this embodiment. The network gateway device 110 communicates with one or more storage servers via a network for data communication or interaction. Figure 1 In the illustrated embodiment, two sets of storage servers are shown: a first storage server 120 and a second storage server 130.

[0050] The gateway device 110 can synchronize data between the first storage server 120 and the second storage server 130 on the other end, thereby enabling data synchronization between the network to which the first storage server 120 belongs and the network to which the second storage server 130 belongs, while ensuring security.

[0051] Optionally, the network to which the first storage server 120 belongs may include multiple servers to form a server cluster for storing data, and the network to which the second storage server 130 belongs may include multiple servers to form a server cluster for storing data.

[0052] In this embodiment, the network gateway device 110 includes a file service module, which is mainly used for downloading files from the front-end first storage server 120, transferring data within the device, and uploading files to the back-end second storage server 130. The network gateway device 110 may also include a scanning module, used to monitor the file directory on the first storage server 120 in real time, detect the creation, deletion, and modification of files on the first storage server 120, and notify the synchronization module to perform read, write, delete, and rename operations based on the changes in attributes.

[0053] The aforementioned network gateway device 110, first storage server 120, and second storage server 130 can be electronic devices with storage and processing functions. For example... Figure 2 The diagram shown is a block illustration of an electronic device. The electronic device 200 may include a memory 211 and a processor 213. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device 200. For example, the electronic device 200 may also include components that are more... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0054] The memory 211 and processor 213 described above are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The processor 213 described above is used to execute executable modules stored in the memory.

[0055] The memory 211 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 211 stores programs, and the processor 213 executes these programs upon receiving execution instructions. The methods executed by the electronic device 200, as defined in any embodiment of this application, can be applied to or implemented by the processor 213.

[0056] The aforementioned processor 213 may be an integrated circuit chip with signal processing capabilities. The processor 213 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0057] The electronic device 200 in this embodiment can be used to execute various steps in the various methods provided in the embodiments of this application. The implementation process of the file scanning method is described in detail below through several embodiments.

[0058] This application provides a document scanning method. The steps of the document scanning method in this embodiment can be achieved by: Figure 1 The network gateway device shown or Figure 2 The electronic device shown performs this action.

[0059] To facilitate understanding of the methodology, the technical terms used in this application will be explained first:

[0060] Network Attached Storage (NAS) is a file-level computer data storage server that connects to a computer network and provides data access to users on heterogeneous networks. It typically uses network file sharing protocols to provide file access. File-level storage is contrasted with block-level storage. Network file sharing protocols can include Network File System (NFS), Server Message Block (SMB), File Transfer Protocol (FTP), and Secret File Transfer Protocol (SFTP). Ragel technology is a state machine compiler primarily used for handling character input and syntax parsing. ZeroMQ (also known as...) ZeroMQ (or ZMQ) is similar to an embeddable network library, but it behaves like a concurrency framework. ZeroMQ provides sockets for transmitting atomic messages across various protocols, including intra-process, inter-process, Transmission Control Protocol (TCP), and multicast. It supports multi-point-to-multi-point socket connections using push, publish-subscribe, task distribution, and request-response modes. The Network File System Program Interface (NFPI) is a development kit specifically for isolation gateways and unidirectional optical gateways, used to connect to various common network protocols and provide a unified operation interface and error code management.

[0061] The following is combined with Figure 3 The document scanning method provided in the embodiments of this application is described.

[0062] Step 310: Obtain the current directory information of the target server through the first-stage scanning process.

[0063] The first-stage scanning process is implemented through the first process.

[0064] The target server can be a network file storage server, which can be used to store data for users to access.

[0065] Before executing step 310, a service connection with the target server can be established first, and then a scanning process can be created: the first process of the first-stage scanning process and the second process of the second-stage scanning process.

[0066] Optionally, a first timer can be created before the first phase of the scanning process. This first timer recursively calls the remote scanning interface in each round to obtain directory information from the target server. For example, this remote scanning interface could be `nfpi::scandir`.

[0067] For example, the scan results of this first-stage scanning process may include file information, directory information, etc.

[0068] The file information may include: file name, file size, modification time, etc., and the directory information may include: directory name, number of subfiles and subdirectories, etc.

[0069] Optionally, this first-stage scanning process can include directory data and file data. The directory tree structure can be used to determine if directories have been added or deleted, and the file index structure can be used to determine if files have been added or deleted.

[0070] For example, the directory data can be presented as a directory tree structure, and the file data can be presented as a file data index structure.

[0071] Step 320: Compare the current directory information with the historical directory information to determine the updated information in the current directory information.

[0072] The historical directory information refers to the latest version of the target server's directory information currently stored locally. This historical directory information may change as the scan progresses.

[0073] This historical directory information can be stored in the local database of the gateway device.

[0074] Optionally, steps 310 and 320 can be processed in parallel. Each time a file or directory information is obtained, the file information can be compared with historical directory information, and the directory information can be compared with historical directory information to determine the updated information in the current directory information.

[0075] Step 330: Transmit the updated information to the second-stage scanning process.

[0076] In this embodiment, the second-stage scanning process can be implemented through a second process.

[0077] Optionally, the first process of the first-stage scanning process can use ZMQ communication to implement message queue communication with the second process of the second-stage scanning process.

[0078] Optionally, the first-stage scanning process can be implemented using a first thread, and the second-stage scanning process can be implemented using a second thread. The first thread and the second thread can also communicate via message queue using ZMQ.

[0079] In the above implementation, different threads or different processes can be used to make program deployment more convenient and flexible.

[0080] Step 340: The second-stage scanning process is used to process the updated information to update the historical directory information and obtain the updated historical directory information.

[0081] For example, updating the historical directory information may include adding new directories, changing the time of directory changes, adding new files, deleting new files, changing file content, etc.

[0082] In this embodiment, steps 310 to 330 described above can be implemented by a first-stage scanning process. Step 340 described above can be implemented by a second-stage scanning process.

[0083] The two-stage scanning process described above allows for preliminary scanning in the first stage to identify data changes, followed by the transmission of updated information to the second stage, thus achieving phased processing. Furthermore, the two scanning stages can be implemented using different processes or threads, enabling parallel processing and improving file scanning efficiency.

[0084] To better leverage the two-stage scanning process, relatively complex update processes can be pushed to the second-stage scanning process, while simpler changes can be completed directly by the first-stage scanning process.

[0085] The current directory information includes multiple file information items. Step 320 above may include comparing each file information item with the content recorded in the historical directory information to determine the file update information.

[0086] File update information can include file additions, file modifications, and file deletions. Therefore, file update information can include information about newly added files and file modifications.

[0087] In one embodiment, the above-mentioned comparison of each file information with the content recorded in the historical directory information to determine the file update information includes: comparing each file information with the content recorded in the historical directory information to determine whether the file corresponding to each file information exists in the historical directory information.

[0088] If the file corresponding to the first file information does not exist in the historical directory information, then the file addition information is determined based on the first file information.

[0089] The first file information is any one of the multiple file information items contained in the current directory information. By comparing each file information item in the current directory information with the historical directory information, it is determined whether the file corresponding to each file information item in the current directory information is newly added.

[0090] For example, by comparing the filenames in each file information with the filenames in the historical directory information, it can be determined whether each file in the current directory information determined from the target server is newly added.

[0091] In one embodiment, the above-mentioned comparison of each file information with the content recorded in the historical directory information to determine the file update information includes: comparing each file information with the content recorded in the historical directory information to determine whether the file corresponding to each file information exists in the historical directory information.

[0092] If the file corresponding to the second file information exists in the historical directory information, and the second file information is different from the file information corresponding to the historical directory information, then the file change information is determined based on the second file information.

[0093] The second file information is any one of the multiple file information items contained in the current directory information.

[0094] For example, the filename of the second file information can be compared with the filenames in the historical directory information to determine whether the file corresponding to the second file information exists in the historical directory information.

[0095] If it is determined that the file corresponding to the second file information exists in the historical directory information, then the file information of the second file information is compared with the file information of the file corresponding to the historical directory information to determine whether the file information of the second file information is the same as the file information of the file corresponding to the historical directory information.

[0096] In this embodiment, if the updated information includes file addition information and file change information, then step 330 can transmit the file addition information and / or file change information to the second-stage scanning process so that the second-stage scanning process can improve the file addition and / or file change.

[0097] Optionally, the historical directory information may include a file tree, which comprises multiple nodes, each representing a directory or a file. This file tree records the directories, subdirectories, files, and other nodes included in the historical directory information.

[0098] Step 320 above may include: comparing the filename in each piece of file information with each node of the file tree to determine whether the file corresponding to each piece of file information belongs to any node in the file tree.

[0099] If the file corresponding to the third file information does not belong to a node in the file tree, the file addition information is determined based on the third file information.

[0100] If the file corresponding to the third file information belongs to a node in the file tree, and the third file information is different from the file information to which the corresponding node in the file tree belongs, then the file change information is determined based on the third file information.

[0101] In this embodiment, if the file corresponding to the third file information belongs to a node in the file tree, the version number of the node in the file tree corresponding to the third file information can be updated.

[0102] The third file information refers to any one of the multiple file information items contained in the current directory information. By comparing each file information item in the current directory information with the file tree in the above manner, it is determined whether any files in the current directory information have been added or modified.

[0103] For example, if the filename in the third file information is the same as the name of one of the nodes in the file tree, the third file information can be compared with the file information corresponding to the node with the same file tree name to determine whether the third file information is the same as the file information of the node with the same file tree name.

[0104] After step 320, the file scanning method may further include: adding a new file node corresponding to the third file information in the file tree, and assigning the third file information to the new file node.

[0105] Optionally, the historical directory information may also include the file tree mentioned above, as well as directory data stored in a local database.

[0106] Therefore, after each update to the file tree, the directory data in the local database is also updated synchronously.

[0107] In addition to the file addition and modification information mentioned above, file deletion and subdirectory updates may also occur. File deletion and subdirectory updates are relatively simple operations and can be performed in the first stage of the scanning process.

[0108] Optionally, the current directory information may include multiple subdirectory information, and the update information may include directory update information. Step 320 above may include: comparing each piece of the subdirectory information with the content recorded in the historical directory information to determine the directory update information.

[0109] After step 320, the file scanning method may further include: updating the directories in the historical directory information according to the directory update information.

[0110] The names of each subdirectory in the current directory information obtained from the target server can be compared with the names of each node in the file tree to determine whether each subdirectory name in the current directory information exists in the file tree.

[0111] If any subdirectory does not exist in the file tree, a new node can be assigned to that file tree, and the subdirectory information can be assigned to the new node as a node for that subdirectory. Optionally, a directory creation hook function can also be called to create the subdirectory in the local database.

[0112] If a subdirectory already exists in the file tree, the node version number can be updated in the file tree. Then, the directory information of the subdirectory is compared with the directory information of the node in the file tree. If there are changes, the directory update hook function is called to update the relevant information of the subdirectory in the local database.

[0113] By using the above method, each subdirectory in the current directory information can be compared with the file tree to determine the addition and modification of subdirectories.

[0114] Furthermore, in the first-stage scanning process, all operations on the file tree also modify the directory data in the local database in the same way, ensuring consistency between the file tree and the directory data in the local database. When comparing with the current directory information, the file tree information can be read only from the file tree's memory. Each time the first-stage scanning process restarts, or after the gateway device restarts, it can first read the directory data from the local database to restore the previous scan state, avoiding duplicate pushes of the same file events and improving data processing efficiency.

[0115] In addition to adding and changing files and directories, there is also the possibility that some files or directories may be deleted from the target server.

[0116] Therefore, after comparing the contents of the current directory information with the file tree, it is also possible to determine whether there are subdirectories or files in the file tree that do not exist in the current directory information. If a file or subdirectory exists only in the file tree but not in the current directory information, it means that the file or subdirectory has been deleted by the target server.

[0117] In one implementation, when comparing a subdirectory or file with the file tree, if the name matches a node name in the file tree, the version number of that node can be updated. Therefore, after each comparison of the current directory information with a node in the file tree, the version number of the node can be updated. If it has been updated, it means the corresponding subdirectory or file exists in the current directory information; if it hasn't been updated, it means the corresponding subdirectory or file does not exist in the current directory information.

[0118] If it is confirmed that a file needs to be deleted, the file deletion hook function can be called to delete the corresponding file in the directory data of the local database; if a subdirectory needs to be deleted, the directory deletion hook function can be called to delete the corresponding subdirectory in the directory data of the local database.

[0119] By processing some relatively simple updates directly in the first-stage scanning process, the workload of the subsequent second-stage scanning process can be reduced, as can the amount of data transmitted between the first and second-stage scanning processes. By improving data processing efficiency through the parallel two stages, the amount of data that needs to be transmitted between them can also be reduced, thereby improving the overall data processing efficiency.

[0120] In the first phase of the scanning process, a large amount of data needs to be scanned from the target server. To improve scanning efficiency, corresponding scanning instances can be set according to the overall data volume, so that the data in the target server can be scanned in parallel through multiple scanning instances.

[0121] In scenarios where the source directory structure is complex, with numerous subdirectories and subfiles, scanning the entire directory on the target server can be time-consuming. Therefore, a parallel scanning method can be provided: by starting N single-scan instances, each instance is responsible for scanning a portion of the directory tree. For example, a single scan of the entire directory might take TimeAll. By starting N parallel scan instances, the scanning time of each instance approaches [the time required for a single scan]. This allows for the discovery of file events on the target server in a shorter time.

[0122] Based on the above analysis, step 310 may include steps 311 and 312.

[0123] Step 311: Determine the first number of scan instances based on the number of nodes in the directory tree of the target server.

[0124] Each scan instance is used to scan at least one node in the directory tree, and a first number of scan instances are used to scan all nodes contained in the directory tree.

[0125] like Figure 4 As shown, Figure 4 An example directory tree of the target server is shown. This example includes a five-level structure, with the third-level structure serving as the basis for partitioning and identifying N scan instances.

[0126] Each scan instance performs a scan on the nodes of both the first and second layer structures.

[0127] For example, a directory hierarchy threshold T of the target server's directory tree can be defined, where, in Figure 4 In the example shown, the level threshold T is set to 3.

[0128] Perform a hash modulo operation on the directory name PathT at level T to determine which scan instance should handle it: Scanid = MOD (1，N)(HASH(Path T )); where Scanid represents the ID number identified by scanning.

[0129] In this embodiment, directory names smaller than the hierarchy threshold T are processed by all scan instances. Furthermore, the filename event Filename{1,T} needs to undergo hash modulo calculation and be assigned to a sub-scan instance for processing: Scanid = MOD (1，N) HASH(Filename) {1，T} )).

[0130] Step 312: For the i-th scan instance, obtain the i-th directory information corresponding to the i-th part node from the target server.

[0131] Wherein, the i-th scan instance is any one of the first number of scan instances, i is a positive integer greater than or equal to 1 and less than or equal to N, N is the first number, and the union of the directory information obtained by scanning each scan instance is the current directory information of the target server.

[0132] Step 320 above may include: comparing the i-th directory information with historical directory information through the i-th scan instance to determine the i-th update information in the i-th directory information.

[0133] Specifically, based on each scan instance, the current directory information is compared with the historical directory information to obtain the updated information in the current directory information.

[0134] By processing data on the target server synchronously through multiple scanning instances, the efficiency of the first-stage scanning process can be improved, thereby increasing the efficiency of scanning data on the target server.

[0135] Considering that the data obtained under different protocols may have different formats, in order to achieve more accurate data scanning, the content scanned from the target server can be converted into the target format to facilitate subsequent comparison with historical directory information. Based on this, step 310 may include: obtaining initial directory information from the target server, extracting the attribute fields contained in each initial field, and recording each attribute field as target format information to obtain the current directory information of the target server.

[0136] The initial directory information includes several initial fields, each of which represents the contents of a file or a directory.

[0137] For example, the initial field can be a string that represents the contents of a file and the contents of a directory.

[0138] For example, Ragel can be used to parse strings. Ragel can scan the input string, match each field of the string with built-in regular expression rules according to the format of the string, and then process it by matching specific states of the state machine. When the scan is finished, the content contained in the string is confirmed.

[0139] This section defines the rules for different platform designs; the Ragel regular expression matching rules vary depending on the platform. For example, file attribute fields differ between Linux and Windows platforms, thus requiring different Ragel regular expression matching rules.

[0140] For example, file attribute fields in Unix include: type permissions, number of hard links, username, user group, file size, timestamp, and file name.

[0141] In one instance, the file attribute field under Unix can be: -rw-r--r--1root root555Mar 2015:56xxx.txt; in another instance, the file attribute field under Unix can be: -rw-r--r--1root root555Dec 20 2020yyy.txt.

[0142] For example, if the file is a symbolic link, it is additionally necessary to parse and obtain the symbolic link name, and then set the file type to symbolic link.

[0143] For example, file attribute fields in MS-DOS include: date, time, type, and file name.

[0144] In one example, a file attribute field in MS-DOS could be: 2021 / 09 / 07 11:15 <dir>.ssh; In another example, the file attribute field in MS-DOS could be: 2021 / 09 / 07 11:23 8 .bash_history

[0145] Through the above analysis, the attribute fields can be extracted.

[0146] Optionally, the attribute field includes a time field. The above-mentioned extraction of the attribute fields contained in the initial field and recording each attribute field as target format information may include: extracting the attribute fields contained in the initial field; if the time field in the attribute field of the first initial field is incomplete, then the time field of the first initial field is corrected to obtain the corrected time, and the corrected time and the attribute field of the first initial field are recorded as target format information.

[0147] The first initial field can be any one of multiple initial fields.

[0148] Different servers support different signaling protocols, leading to variations in the display format and content of file and directory attribute fields across different servers. The differences in timestamps are particularly noticeable. This can cause inconsistencies in file timestamps when using Ragel parsing, resulting in abnormal scan results.

[0149] For example, there are various types of FTP servers, including IIS server, Filezillia server, and ServU server on the Windows platform, and VsFTPd server, Proftpd server, and Pureftpd server on the Linux platform.

[0150] If the target server supports MLSD signaling, timestamp repair is not necessary. If the target server supports MDTM signaling, the timestamp of the parsed file needs to be corrected. The correction process is as follows: Loop through each time field of the scanned directory. If the current time field is a file's time field, do not correct the timestamp. Continue looping to identify the time field of the next initial field.

[0151] Among them, the missing time field mainly includes two situations: missing year and missing hour and minute.

[0152] Time correction for missing years: If the current initial field in the loop is the initial field of the directory, a threshold time can be set, for example, January 1, 1971. The current time field is compared to this threshold time. If it is less than the threshold time, the month of the current directory is obtained based on the current time field `st_mtime`, and then the current year is obtained based on the current month of the directory. The year field value is then checked; if it is 0, MDTM signaling is executed to modify the timestamp of the directory, and the timestamp in the response data obtained from the MDTM command is parsed and corrected. For example, if the target server supports MLSD commands, the seconds field is set to 0; if the current service is IIS, the current time field `st_mtime` can be corrected using UTC time; otherwise, CST time correction is used. The corrected time field is then used to modify the initial field representing the attributes of the current directory.

[0153] Time correction for missing time and minutes: If the current directory's time field is greater than a preset threshold time, check if the remainder of the current directory's time field `st_mtime` divided by 3600 is 0. If it is 0, the time field can be the number of minutes between the current time and the threshold time (which is 0). If the remainder is 0, it means both the hour and minute fields are zero. Then, execute MDTM signaling to modify the directory's timestamp, parse the timestamp in the response data obtained from the MDTM command, and correct the year, hour, and minute. For example, if the target server supports the MLSD command, the seconds field is set to 0; if the current service is IIS, the current time field `st_mtime` can be corrected using UTC time; otherwise, CST time correction is used. Finally, use the corrected time field to modify the initial field representing the current directory's attributes.

[0154] The above method can make the determined current directory information more accurate. As described above, there may be many types of target servers, and different servers support different signaling, and different signaling processes have different efficiencies. Before executing step 310, it may also include: obtaining the signaling types supported by the target server; if the target server supports at least two signaling types, then determining the target signaling according to the priority of the at least two signaling types.

[0155] The target signaling is used to perform the steps of obtaining initial directory information from the target server and extracting the attribute fields contained in each initial field.

[0156] Of course, if the target server only supports one type of signaling, then the signaling supported by the target server can be used as the target signaling.

[0157] For example, the priorities of different signaling can be preset. For instance, for retrieving file information for a single file, the signaling supported by each server may include: MLST signaling, SIZE+MDTM signaling, LIST signaling, and MLSD signaling. The priorities of each signaling, from highest to lowest, are: MLST signaling, SIZE+MDTM signaling, LIST signaling, and MLSD signaling.

[0158] Among these methods, obtaining file information via MLST is the most efficient. This involves: executing MLST extended signaling and obtaining the corresponding response data; and executing the MLSD signaling parsing process and obtaining the corresponding file information.

[0159] The SIZE+MDTM method retrieves attributes, which has moderate efficiency. This method involves: 1) cyclically executing the SIZE extended signaling. If the SIZE signaling fails, the current error code is determined to be ENOENT. If the current error code is not ENOENT, the LIST process (Ragel parsing) is performed to obtain the file information. 2) If the SIZE extended signaling succeeds, and a network reconnection occurs, the last line of the multi-line response is retrieved. If the last line does not contain the correct error code, the SIZE extended signaling is repeated to retrieve the size information. 3) Then, the MDTM signaling is executed to correct the time field. If MDTM fails and the system error code is not ENOENT, the LIST process (Ragel parsing) is performed to obtain the file information, and the modified file information is concatenated. If MDTM fails, the STAT command is executed, the STAT result is parsed using Ragel, and the modified file information is concatenated.

[0160] Among these methods, the LIST or MLSD signaling used to obtain attributes is the least efficient. If the target server supports MLSD extended signaling, the MLSD signaling is executed, followed by Ragel parsing; otherwise, the standard LIST signaling is executed, followed by Ragel parsing, and finally, the modified file information is concatenated.

[0161] For example, for obtaining file information from batches of files, the signaling supported by each server may include: LIST signaling and MLSD signaling.

[0162] The process of obtaining file information for batch files via LIST signaling can include: first, performing the Ragel::LIST parsing process; when the state machine receives the initial fields of each file or directory, performing regular expression matching based on each attribute field of the initial fields; if each field matches, executing the corresponding state machine to obtain the corresponding key value, and finally obtaining the list of each file.

[0163] The process of obtaining file information for batch files via MLSD signaling can include: first, performing a Ragel::MLSD parsing process; when the state machine receives the initial fields of each file or directory, performing regular expression matching based on each attribute field of the initial fields; if each field matches, executing the corresponding state machine to obtain the corresponding key value, and finally obtaining the list of each file.

[0164] By adaptively selecting the signaling supported by the server to perform step 310 above, the accuracy and effectiveness of obtaining the current directory information can be achieved.

[0165] For example, the aforementioned signaling can be FTP signaling, which is compatible with mainstream servers such as IIS, Filezillia, and ServU on Windows platforms, and VsFTPd, Proftpd, and Pureftpd on Linux platforms. The FTP signaling RFC specification defines standard and extended signaling, and the degree of support for extended signaling varies across different servers. This embodiment utilizes adaptive signaling technology to prioritize high-efficiency signaling to complete the required processing flow, while also adapting to various mainstream servers, thereby improving processing efficiency while meeting data processing requirements.

[0166] In the second stage of the scanning process, there may be a relatively large number of files to be processed. If the method of obtaining information from each file individually is used, the acquisition efficiency may be low.

[0167] Step 340 above may include steps 341 to 345.

[0168] Step 341: Determine the set of files to be processed based on the received file update information.

[0169] The set of files to be processed includes files that can be found in the update information determined through step 320 above.

[0170] Step 342: Determine the directory to which each file belongs in the set of files to be processed belongs, so as to determine the directory to which each file belongs as the set of directories to be processed, and determine the number of files to be processed contained in each directory in the set of directories to be processed.

[0171] Step 343: For the first directory to be processed, determine the method of obtaining file information based on the number of files to be processed contained in the first directory to be processed.

[0172] The first directory to be processed is any one of the directories in the set of directories to be processed. If the number of files to be processed is greater than the specified number, the file information of the files to be processed is obtained using a batch retrieval method; if the number of files to be processed is not greater than the specified number, the file information of the files to be processed is obtained using a single retrieval method. The specified number can be a value such as 100 or 150.

[0173] Step 344: Obtain the file information of each file to be processed from the target server in sequence according to the file acquisition method.

[0174] Step 345: Update the historical directory information based on the file information of each file to be processed.

[0175] The following example illustrates the processing flow of steps 341 to 345 above:

[0176] For example, the set of files to be processed can be stored in a queue of files to be processed. The latest attributes of the files in the queue can be updated periodically, and information about individual files on the target server can be obtained via nfpi::stat. When the number of files to be processed in the set is large, multiple round-trip requests will occur.

[0177] This solution makes the following optimizations: The directory information in the queue of files to be processed is merged and counted. If there are M files to be processed, then the set of files to be processed can be represented as: {Reg1, Reg2, Reg3...RegM}.

[0178] The directory information can be used to identify the K directories to which the file set to be processed belongs, as well as the number of subfiles in each directory: {Dir1, Dir2, Dir3…DirK}, {Dnum1, Dnum2, Dnum3…DnumK}.

[0179] Each directory to be processed is checked. If the number of subfiles Dnumi in the directory exceeds the set threshold, nfpi::scandir is called to obtain file information in batch mode; if the number of files Dnumj in the directory is less than or equal to the threshold, nfpi::stat is called to obtain information for a single file in single mode.

[0180] The above processing method can effectively save network overhead and improve file scanning efficiency.

[0181] Considering that the target server may contain a large amount of content, not all files need to be scanned. Therefore, a filtering strategy can be set to filter out files or directories that do not need to be scanned. Based on this, step 310 above may include: obtaining current directory information from the target server according to a preset filtering strategy.

[0182] The filtering strategy includes one or more of the following: directory filtering strategy, file filtering strategy, and file priority strategy.

[0183] Directory filtering strategies can filter out directories that are not needed in the business process, thus preventing scanning of the filtered directories. Scanning using directory filtering strategies can reduce the overall scanning time overhead.

[0184] For example, directory filtering strategies can be configured using directory name regular expressions, wildcard rules, etc.

[0185] File filtering strategies can be used to limit the selection of files of interest or exclusion. Configuration methods include file blacklists and whitelists. These can be configured using filename regular expressions, wildcard rules, file size, file modification time, file type, and file content.

[0186] File priority strategies can be implemented using an emitter, allowing files of interest to be scanned and processed first. For example, a collection of files in the same parent directory might use quicksort to ensure proper sorting during scanning; while the emitter might use merge sort to reorder the batch of files to be sent. Exemplarily, file priority can be set by sorting by filename, file size, or file modification time.

[0187] Considering the different types of target servers, the file sharing protocols they support may also differ. Therefore, based on the different file sharing protocols supported by the target server, before step 310, the file scanning method may further include: obtaining the target type of the file sharing protocol supported by the target server, and determining the scanning plugin corresponding to the target type based on the target type.

[0188] The file sharing protocol includes at least one of the following: NFS, SMB, FTP, and SFTP protocols. The scanning plugin is used to execute both the first-stage and second-stage scanning processes.

[0189] For example, URLs can be constructed based on necessary information such as the target server's service protocol, server address, server port, username, password, and working directory.

[0190] Then, scanning plugins can be dynamically loaded based on the target server's file sharing protocol: this solution is designed to load different protocol plugins according to the protocol category. This way, only the necessary dynamic library protocol interfaces are loaded for the current connection instance, effectively reducing the instance's runtime memory. Furthermore, it can support extensions to other protocols in the future; simply implementing the corresponding interface plugin will allow integration into the existing framework.

[0191] Among them, the NFS protocol plugin supports two protocol versions, NFSv3 and NFSv4; the SMB plugin supports protocols such as SMB1 (LANMAN2, NT1, CIFS), SMB2, and SMB3; the FTP plugin supports FTP, FTP explicit SSL / TLS, and FTP implicit SSL / TLS protocols; and the SFTP plugin supports protocols such as SFTP, SSH, and SCP.

[0192] After completing the first and second phases of the scanning process, you can uninstall the scanning plugin: close the connection and free up memory space.

[0193] For example, an NFPI network operation interface can be provided, supporting unified signaling access for SMB, NFS, SFTP, and FTP protocols, and providing a UNIX-like interface style. Based on this, the upper-layer scanning component layer only needs to focus on the logical organization of the scan, without needing to concern itself with the specific signaling operations within NFPI.

[0194] In one instance, a single protocol is treated as a separate plugin library, requiring the protocol plugin interface to implement the following table:

[0195]

[0196] To increase the success rate of file scanning, various tests can be performed on the target server before step 310.

[0197] Optionally, connectivity with the target server can be tested.

[0198] For example, you can test the address and port to check if a connection can be established normally with the target server; then test the username and password to check if the logged-in user's account and password are correct.

[0199] Optionally, the permissions of the target server can be tested. These permissions include one or more of the following: directory creation, directory deletion, file creation, file deletion, file read operations, and file write operations.

[0200] For example, you can test directory creation permissions: check if the target server supports directory creation; you can test file creation permissions: check if the target server supports file creation; you can test file write permissions: check if the target server supports file write operations; you can test file read permissions: check if the target server supports file read operations; you can test file deletion permissions: check if the target server supports deleting files; you can test directory deletion permissions: check if the target server can delete directories.

[0201] Optionally, the scalability of the target server can be tested. This scalability includes whether it supports modifying file attributes.

[0202] For example, you can test file attribute acquisition: check if the target server can normally obtain some basic file attributes; you can test timestamp modification permissions: check if the target server supports modifying file timestamps; you can test long filename creation: check if the target server supports creating long filenames; you can test file fragment permissions: check if the target server supports file offset operations.

[0203] Optionally, if any test fails, the reason for the failure will be output. Optionally, if the test succeeds, a success message will also be output. For example, the test results can be displayed in SYSLOG or WEBUI format.

[0204] The file scanning method provided in this application embodiment achieves file scanning by remotely connecting to a target server via a network. It can utilize the NFPI development kit, which supports FTP, SFTP, SMB, and NFS standard protocols, and provides logically orchestrated scanning components and services on top of the NFPI development kit.

[0205] In this embodiment, the development work of the protocol processing layer and the scanning logic layer can be effectively isolated through the NFPI unified interface, and the protocol extension can be flexibly supported.

[0206] Furthermore, in the embodiments of this application, the Ragel parser is used, which significantly improves parsing efficiency and enhances the scanning interface in the scenario of a list of 100,000 files. Compared with conventional correction, the FTP timestamp correction technology can reduce signaling interactions and improve processing speed in the case of a list of 100,000 files.

[0207] Based on the same application concept, this application also provides a document scanning device corresponding to the document scanning method. Since the principle of the device in this application is similar to that of the aforementioned document scanning method, the implementation of the device in this application can refer to the description in the above method embodiments, and the repeated parts will not be described again.

[0208] Please see Figure 5 This is a functional module diagram of the file scanning device provided in this application embodiment. Each module in the file scanning device in this embodiment is used to execute the steps in the above method embodiments. The file scanning device includes: a first acquisition module 410, a comparison module 420, a transmission module 430, and a first update module 440; the contents of each module are as follows: The first acquisition module 410 is used to acquire the current directory information of the target server through a first-stage scanning process; the comparison module 420 is used to compare the current directory information with historical directory information to determine the updated information in the current directory information, wherein the historical directory information is the latest version of the target server's directory information currently stored locally; the transmission module 430 is used to transmit the updated information to a second-stage scanning process; the first update module 440 is used to process the updated information through the second-stage scanning process to update the historical directory information to obtain updated historical directory information.

[0209] In one possible implementation, the current directory information includes multiple file information items, and the update information includes file update information. The comparison module 420 is used to compare each file information item with the content recorded in the historical directory information to determine the file update information.

[0210] In one possible implementation, the file update information includes: file addition information; comparison module 420 is used to: compare each file information with the content recorded in the historical directory information to determine whether the file corresponding to each file information exists in the historical directory information; if the file corresponding to the first file information does not exist in the historical directory information, then determine the file addition information based on the first file information, wherein the first file information is any one of the multiple file information contained in the current directory information.

[0211] In one possible implementation, the file update information includes: file change information; a comparison module 420, used to compare each file information with the content recorded in the historical directory information to determine whether the file corresponding to each file information exists in the historical directory information; if the file corresponding to the second file information exists in the historical directory information, and the second file information is different from the file information corresponding to the historical directory information, then the file change information is determined based on the second file information, wherein the second file information is any one of the multiple file information contained in the current directory information.

[0212] In one possible implementation, the historical directory information may include: a file tree, which includes multiple nodes, each node representing a directory or a file; a comparison module 420, used to compare the filenames in each piece of file information with each node of the file tree to determine whether the file corresponding to each piece of file information belongs to any node in the file tree; if the file corresponding to the third piece of file information does not belong to any node in the file tree, file addition information is determined based on the third piece of file information, wherein the third piece of file information is any one of the multiple pieces of file information contained in the current directory information; if the file corresponding to the third piece of file information belongs to a node in the file tree, and the third piece of file information is different from the file information to which the corresponding node in the file tree belongs, then file change information is determined based on the third piece of file information.

[0213] In one possible implementation, if the file corresponding to any third file information does not belong to a node in the file tree, the file scanning device provided in this application embodiment may further include: an addition module, used to add a new file node corresponding to the third file information in the file tree, and assign the third file information to the new file node.

[0214] In one possible implementation, the first acquisition module 410 is configured to: determine a first number of scan instances based on the number of nodes in the directory tree of the target server, wherein each scan instance is used to scan at least one node in the directory tree, and the first number of scan instances are used to scan the nodes contained in the directory tree; for the i-th scan instance, obtain the i-th directory information corresponding to the i-th part of the nodes from the target server, wherein the i-th scan instance is any one of the first number of scan instances, i is a positive integer greater than or equal to 1 and less than or equal to N, and N is the first number, wherein the union of the directory information obtained based on each scan instance is the current directory information of the target server; the comparison module 420 is configured to compare the i-th directory information with historical directory information through the i-th scan instance to determine the i-th update information in the i-th directory information, wherein the current directory information is compared with historical directory information based on each scan instance to obtain the update information in the current directory information.

[0215] In one possible implementation, the first acquisition module 410 includes an information acquisition unit and a field extraction unit; the information acquisition unit is used to acquire initial directory information from the target server, the initial directory information including multiple initial fields, each initial field being used to represent the content of a file or the content of a directory; the field extraction unit is used to extract the attribute fields contained in each initial field and record each attribute field as target format information to obtain the current directory information of the target server.

[0216] In one possible implementation, the attribute field includes a time field; a field extraction unit is used to extract the attribute fields contained in each of the initial fields; if the time field in the attribute field of the first initial field is incomplete, the time field of the first initial field is corrected to obtain a corrected time, wherein the first initial field is any one of the multiple initial fields; the corrected time and the attribute field of the first initial field are recorded as target format information.

[0217] In one possible implementation, the file scanning device may further include: a second acquisition module for acquiring the signaling types supported by the target server; and a first determination module for determining a target signaling based on the priority of at least two signaling types if the target server supports at least two signaling types, wherein the target signaling is used to execute the aforementioned information acquisition unit and field extraction unit.

[0218] In one possible implementation, the current directory information includes: multiple subdirectory information, and the update information includes: directory update information; the comparison module 420 is used to compare each of the subdirectory information with the content recorded in the historical directory information to determine the directory update information; the file scanning device may further include: a second update module, used to update the directories in the historical directory information according to the directory update information.

[0219] In one possible implementation, the update information includes file update information;

[0220] The first update module 440 is configured to: determine a set of files to be processed based on the received file update information; determine the directories to which each file belongs in the set of files to be processed belongs, thereby defining the directories to which each file belongs as a set of directories to be processed, and determining the number of files to be processed contained in each directory in the set of directories to be processed; for a first directory to be processed, determine a file information acquisition method based on the number of files to be processed contained in the first directory to be processed, wherein the first directory to be processed is any directory in the set of directories to be processed; if the number of files to be processed is greater than a specified number, use a batch acquisition method to acquire the file information of the files to be processed; if the number of files to be processed is not greater than the specified number, use a single acquisition method to acquire the file information of the files to be processed; acquire the file information of each file to be processed sequentially from the target server according to the file acquisition method; and update the historical directory information based on the file information of each file to be processed.

[0221] In one possible implementation, the first-stage scanning process is implemented through a first process; the second-stage scanning process is implemented through a second process.

[0222] In one possible implementation, the first acquisition module 410 is used to acquire current directory information from the target server according to a preset filtering strategy, wherein the filtering strategy includes one or more of the following: directory filtering strategy, file filtering strategy, and file priority strategy.

[0223] In one possible implementation, the file scanning device may further include: a third acquisition module, configured to acquire the target type of the file sharing protocol supported by the target server, wherein the file sharing protocol includes at least one of the following: NFS protocol, SMB protocol, FTP protocol, and SFTP protocol; and a second determination module, configured to determine the scanning plugin corresponding to the target type based on the target type, wherein the scanning plugin is used to execute the first-stage scanning process and the second-stage scanning process.

[0224] In one possible implementation, the file scanning device may further include: a first testing module for testing connectivity with the target server; a second testing module for testing permissions of the target server, wherein the permissions include one or more of the following: directory creation, directory deletion, file creation, file deletion, file read operation, and file write operation; a third testing module for testing the scalability of the target server, wherein the scalability includes whether it supports modifying file attributes; and an output module for outputting the reason for test failure if any test fails.

[0225] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the file scanning method described in the above method embodiments.

[0226] The computer program product of the file scanning method provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the file scanning method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0227] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0228] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0229] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0230] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0231] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / dir>

Claims

1. A document scanning method, characterized in that, include: The first phase of the scanning process obtains the current directory information of the target server; The current directory information is compared with the historical directory information to determine the updated information in the current directory information, wherein the historical directory information is the directory information of the latest version of the target server currently stored locally; The updated information is transmitted to the second-stage scanning process; The second-stage scanning process processes the updated information to update the historical directory information, obtaining updated historical directory information. This includes: determining a set of files to be processed based on the received file update information; determining the directories to which each file belongs in the set of files to be processed belongs, thus defining the directories to which each file belongs as a set of directories to be processed, and determining the number of files to be processed contained in each directory in the set of directories to be processed; for a first directory to be processed, determining a file information acquisition method based on the number of files to be processed contained in the first directory to be processed, wherein the first directory to be processed is any one of the directories to be processed in the set of directories to be processed; if the number of files to be processed is greater than a specified number, using a batch acquisition method to acquire the file information of the files to be processed; if the number of files to be processed is not greater than a specified number, using a single acquisition method to acquire the file information of the files to be processed; sequentially acquiring the file information of each file to be processed from the target server according to the file information acquisition method; and updating the historical directory information based on the file information of each file to be processed.

2. The method according to claim 1, characterized in that, The current directory information includes: multiple file information, and the update information includes: file update information; The step of comparing the current directory information with historical directory information to determine the updated information in the current directory information includes: The file information is compared with the contents recorded in the historical directory information to determine the file update information.

3. The method according to claim 2, characterized in that, The file update information includes: newly added file information; The step of comparing each of the file information entries with the content recorded in the historical directory information to determine the file update information includes: The file information is compared with the contents recorded in the historical directory information to determine whether the file corresponding to each file information exists in the historical directory information; If the file corresponding to the first file information does not exist in the historical directory information, then the file addition information is determined based on the first file information, wherein the first file information is any file information among the multiple file information contained in the current directory information.

4. The method according to claim 2, characterized in that, The file update information includes: file change information; The step of comparing each of the file information entries with the content recorded in the historical directory information to determine the file update information includes: The file information is compared with the contents recorded in the historical directory information to determine whether the file corresponding to each file information exists in the historical directory information; If the file corresponding to the second file information exists in the historical directory information, and the second file information is different from the file information corresponding to the historical directory information, then the file change information is determined based on the second file information, wherein the second file information is any file information among the multiple file information contained in the current directory information.

5. The method according to claim 2, characterized in that, The historical directory information includes: a file tree, which includes multiple nodes, each node representing a directory or a file; The step of comparing each of the file information entries with the content recorded in the historical directory information to determine the file update information includes: The filenames in each of the file information items are compared with each node of the file tree to determine whether the file corresponding to each of the file information items belongs to any node in the file tree; If the file corresponding to the third file information does not belong to a node in the file tree, the file addition information is determined based on the third file information, wherein the third file information is any file information among the multiple file information contained in the current directory information; If the file corresponding to the third file information belongs to a node in the file tree, and the third file information is different from the file information to which the corresponding node in the file tree belongs, then the file change information is determined based on the third file information.

6. The method according to claim 5, characterized in that, If the file corresponding to any third file information does not belong to a node in the file tree, the method further includes: A new file node corresponding to the third file information is added to the file tree, and the third file information is assigned to the new file node.

7. The method according to claim 1, characterized in that, The step of obtaining the current directory information of the target server includes: Based on the number of nodes in the directory tree of the target server, a first number of scan instances are determined, wherein each scan instance is used to scan at least one node in the directory tree, and the first number of scan instances are used to scan the nodes contained in the directory tree. For the i-th scan instance, obtain the i-th directory information corresponding to the i-th part node from the target server, wherein the i-th scan instance is any one of the first number of scan instances, i is a positive integer greater than or equal to 1 and less than or equal to N, N is the first number, and the union of the directory information obtained based on each scan instance is the current directory information of the target server. The step of comparing the current directory information with historical directory information to determine the updated information in the current directory information includes: Through the i-th scan instance, the i-th directory information is compared with the historical directory information to determine the i-th update information in the i-th directory information. Specifically, based on each scan instance, the current directory information is compared with the historical directory information to obtain the update information in the current directory information.

8. The method according to claim 1, characterized in that, The step of obtaining the current directory information of the target server includes: Obtain initial directory information from the target server. The initial directory information includes multiple initial fields, each of which represents the content of a file or the content of a directory. Extract the attribute fields contained in each of the initial fields, and record each attribute field as target format information to obtain the current directory information of the target server.

9. The method according to claim 8, characterized in that, The attribute fields include a time field; The step of extracting the attribute fields contained in each of the initial fields and recording each attribute field as target format information includes: Extract the attribute fields contained in the initial fields of each item; If the time field in the attribute field of the first initial field is incomplete, the time field of the first initial field is corrected to obtain the corrected time, wherein the first initial field is any one of multiple initial fields; Record the correction time and the attribute fields of the first initial field as target format information.

10. The method according to claim 8, characterized in that, Before obtaining the initial directory information from the target server, the method further includes: Obtain the signaling types supported by the target server; If the target server supports at least two signaling types, then the target signaling is determined according to the priority of the at least two signaling types, wherein the target signaling is used to perform the steps of obtaining initial directory information from the target server and extracting the attribute fields contained in each of the initial fields.

11. The method according to claim 1, characterized in that, The current directory information includes: multiple subdirectory information, and the update information includes: directory update information; The step of comparing the current directory information with historical directory information to determine the updated information in the current directory information includes: The information in each subdirectory is compared with the content recorded in the historical directory information to determine the directory update information; The method further includes: The directories in the historical directory information are updated according to the directory update information.

12. The method according to any one of claims 1-11, characterized in that, The first stage of the scanning process is implemented through the first process; The second-stage scanning process is implemented through a second process.

13. The method according to any one of claims 1-11, characterized in that, The step of obtaining the current directory information of the target server includes: According to a preset filtering strategy, the current directory information is obtained from the target server, wherein the filtering strategy includes one or more of the following: directory filtering strategy, file filtering strategy, and file priority strategy.

14. The method according to any one of claims 1-11, characterized in that, Before obtaining the current directory information of the target server, the method further includes: Obtain the target type of file sharing protocol supported by the target server, wherein the file sharing protocol includes at least one of the following: NFS protocol, SMB protocol, FTP protocol, and SFTP protocol; Based on the target type, a scanning plugin corresponding to the target type is determined, wherein the scanning plugin is used to execute the first stage scanning process and the second stage scanning process.

15. The method according to any one of claims 1-11, characterized in that, Before obtaining the current directory information of the target server, the method further includes at least one of the following: Test the connectivity with the target server; The permissions of the target server are tested, wherein the permissions include one or more of the following: directory creation, directory deletion, file creation, file deletion, file read operation, and file write operation; The scalability of the target server is tested, wherein the scalability includes whether it supports modifying file attributes; If any test fails, output the reason for the failure.

16. A document scanning device, characterized in that, include: The first acquisition module is used to obtain the current directory information of the target server through the first-stage scanning process; The comparison module is used to compare the current directory information with the historical directory information to determine the updated information in the current directory information, wherein the historical directory information is the directory information of the latest version of the target server currently stored locally; The transmission module is used to transmit the updated information to the second-stage scanning process; The first update module is used to process the update information through the second-stage scanning process to update the historical directory information and obtain updated historical directory information. The first update module is used to determine the set of files to be processed based on the received file update information; The directories to which each file belongs in the set of files to be processed are determined, so as to define the directories to which each file belongs as a set of directories to be processed, and to determine the number of files to be processed contained in each directory in the set of directories to be processed. For the first directory to be processed, the method for obtaining file information is determined based on the number of files to be processed contained in the first directory to be processed, wherein the first directory to be processed is any one of the directories to be processed in the set of directories to be processed; if the number of files to be processed is greater than a specified number, the batch acquisition method is used to obtain the file information of the files to be processed; if the number of files to be processed is not greater than the specified number, the single acquisition method is used to obtain the file information of the files to be processed. According to the file information acquisition method, the file information of each file to be processed is sequentially obtained from the target server, and the historical directory information is updated according to the file information of each file to be processed.

17. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores machine-readable instructions executable by the processor, wherein when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 15.

Images