Repeated file scanning method and device, equipment and storage medium

Abstract

The invention discloses a duplicate file scanning method and device, equipment and a storage medium, and relates to the technical field of file processing, a structured index is established through metadata independent of actual content of a file, only a small part of system files need to be subjected to full-quantity verification in subsequent duplicate file identification, and the scanning cost of the duplicate file is greatly reduced. The method comprises the steps that file metadata meeting query conditions are obtained through a file attribute database, file attribute data are independent of actual file content and used for recording a structured data set of file description information, and the file metadata are structured description information recorded by the file attribute database for each file; according to the file metadata meeting the query conditions, a structured index database is established, each record in the structured index database is stored in a set field combination formatting mode, and all the records are stored in order through a tree structure; and carrying out multi-layer progressive inspection on the system file according to the structured index database so as to identify and obtain duplicate files in the system.

Description

Technical Field

[0001] This application relates to the field of document processing technology, and in particular to a method, apparatus, device and storage medium for scanning duplicate documents. Background Technology

[0002] Duplicate file scanning is the process of identifying redundant data by comparing file content or feature values. Its core function is to free up storage space and improve file management efficiency. It is widely used in scenarios such as disk cleanup and data backup.

[0003] In related technologies, duplicate file scanning mainly relies on full hash calculation. This process requires traversing the file system to read the contents of each file one by one and calculating a complete hash value for each file, such as MD5 or SHA, to determine file duplication. However, with the increase in storage capacity and the surge in the number of files, full hash calculation faces problems such as excessive scanning time and high CPU / disk I / O resource consumption. Especially in scenarios involving large file processing and frequent scanning, it will repeatedly consume resources, resulting in resource waste. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, device and storage medium for scanning duplicate files. The main purpose is to solve the problems of excessive scanning time and high CPU / disk I / O resource consumption in the full hash calculation in the prior art, especially in the case of large file processing and frequent scanning, which will repeatedly consume resources and cause resource waste.

[0005] According to a first aspect of this application, a method for scanning duplicate documents is provided, comprising: The file metadata that meets the query conditions is obtained through the file attribute database. The file attribute data is unique to the actual content of the file and is a structured data set used to record file description information. The file metadata is the structured description information recorded by the file attribute database for each file. Based on the file metadata that meets the query conditions, a structured index library is established. Each record in the structured index library is stored in a formatted way with a set combination of fields, and all records are stored in an ordered manner through a tree structure. The system files are subjected to multi-level progressive verification based on the structured index library to identify duplicate files in the system.

[0006] Furthermore, the step of obtaining file metadata that meets the query conditions through the file attribute database includes: Open the disk volume in the file attribute database using system functions to obtain a volume handle with read permissions; The device control function is invoked to initiate a query request to the file attribute database, so as to pass the volume handle with read permission and the pre-configured file query parameters to the file attribute database through the query request; The continuous structures output from the file attribute database in the buffer are parsed in batches to obtain file metadata that meets the query conditions. Each structure corresponds to a file metadata record.

[0007] Furthermore, the batch parsing of the continuous structure output from the file attribute database in the buffer to obtain file metadata that meets the query conditions includes: Starting from the beginning position of the file attribute data output in the buffer, traverse and parse each structure output in the buffer until all structures output in the buffer are traversed to obtain the file metadata; The parsed file metadata is filtered using pre-defined query criteria to obtain file metadata that meets the query criteria.

[0008] Furthermore, after the method establishes a structured index based on the file metadata that meets the query conditions, the method further includes: When the file system changes, the structured index is logically updated using the metadata change records captured by the file attribute database. If the file system does not support setting change records, the modified attributes of the current file are used as a combination of features reflecting the file status, and the structured index is logically updated when it is determined that the metadata has changed.

[0009] Furthermore, before performing multi-level progressive checks on system files based on the structured index library to identify duplicate files in the system, the method further includes: A multi-layered progressive verification is pre-constructed, which includes at least file byte verification, partial hash verification, and full verification based on the partial hash verification results.

[0010] Furthermore, the step of performing multi-level progressive checks on system files based on the structured index library to identify duplicate files in the system includes: Based on the structured index, statistical characteristics of system files based on the number of bytes are extracted, and system files with the same number of bytes based on the statistical characteristics are used as the first-level verification files; In the first-level verification file, the sampled content of each system file is read one by one, and the system file with a consistent hash value of the sampled content is used as the second-level verification file; In the second-level verification file, the complete content of each system file is read one by one, and system files with consistent hash values ​​in their complete content are identified as duplicate files in the system.

[0011] Further, after performing multi-level progressive checks on system files based on the structured index library to identify duplicate files in the system, the process includes: The hash value obtained by multi-level progressive verification is associated and bound with the file path to form a mapping relationship; In response to a call to the hash value of a system file to be queried, the hash value of the system file to be queried is located based on the file path in the mapping relationship.

[0012] According to a second aspect of this application, a duplicate document scanning apparatus is provided, comprising: The acquisition unit is used to acquire file metadata that meets the query conditions through the file attribute database. The file attribute data is unique to the actual content of the file and is a structured data set used to record file description information. The file metadata is the structured description information recorded by the file attribute database for each file. The establishment unit is used to establish a structured index library based on the file metadata that meets the query conditions. Each record in the structured index library is stored in a formatted combination of set fields, and all records are stored in an ordered manner through a tree structure. The identification unit is used to perform multi-level progressive checks on system files based on the structured index library to identify duplicate files in the system.

[0013] Further, the acquisition unit includes: The acquisition module is used to open the disk volume in the file attribute database through system functions in order to obtain a volume handle with read permissions; The initiation module is used to call the device control function to initiate a query request to the file attribute database, so as to pass the volume handle with read permission and the pre-configured file query parameters to the file attribute database through the query request; The parsing module is used to parse the continuous structures output by the file attribute database in the buffer in batches to obtain file metadata that meets the query conditions. Each structure corresponds to a file metadata record.

[0014] Furthermore, the parsing module is specifically used for: Starting from the beginning position of the file attribute data output in the buffer, traverse and parse each structure output in the buffer until all structures output in the buffer are traversed to obtain the file metadata; The parsed file metadata is filtered using pre-defined query criteria to obtain file metadata that meets the query criteria.

[0015] Furthermore, the matching unit includes: The update unit is used to logically update the structured index library by capturing metadata change records from the file attribute database when the file system changes after the structured index library is established based on the file metadata that meets the query conditions. The update unit is further configured to, if the file system does not support the setting of change records, use the modified attributes of the current file as a feature combination reflecting the file status, and logically update the structured index library when it is determined that the metadata has changed.

[0016] Furthermore, the device also includes: The construction unit is used to pre-construct a multi-layer progressive verification before performing multi-layer progressive verification on system files according to the structured index library to identify duplicate files in the system. The multi-layer progressive verification includes at least file byte verification, partial hash verification, and full verification based on the partial hash verification results.

[0017] Furthermore, the identification unit is specifically used for: Based on the structured index, statistical characteristics of system files based on the number of bytes are extracted, and system files with the same number of bytes based on the statistical characteristics are used as the first-level verification files; In the first-level verification file, the sampled content of each system file is read one by one, and the system file with a consistent hash value of the sampled content is used as the second-level verification file; In the second-level verification file, the complete content of each system file is read one by one, and system files with consistent hash values ​​in their complete content are identified as duplicate files in the system.

[0018] Furthermore, the device also includes: The binding unit is used to associate and bind the hash value calculated by the multi-level progressive verification with the file path after the system files are identified by performing multi-level progressive verification according to the structured index library to identify duplicate files in the system, thereby forming a mapping relationship. The locating unit is used to locate the hash value of the system file to be queried in response to the hash value call of the system file to be queried, based on the file path in the mapping relationship.

[0019] According to a third aspect of this application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0020] According to a fourth aspect of this application, a readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0021] By employing the above technical solution, this application provides a method, apparatus, device, and storage medium for scanning duplicate files. Compared with existing technologies that rely on full hash calculations to achieve duplicate file scanning, this application obtains file metadata that meets query conditions from a file attribute database. The file attribute data is independent of the actual file content and is a structured data set used to record file description information. The file metadata is the structured description information recorded by the file attribute database for each file. Based on the file metadata that meets the query conditions, a structured index is established. Each record in the structured index is formatted and stored using a set combination of fields, and all records are stored in an ordered tree structure. Multi-level progressive checks are performed on system files based on the structured index to identify duplicate files in the system. The entire process establishes a structured index using metadata independent of the actual file content, ensuring accurate association between index records and system files, improving system file reading efficiency. Subsequent duplicate file identification uses multi-level verification coverage to adapt to different scenarios. While ensuring the accuracy of duplicate file identification results, only a small portion of system files need full verification, greatly reducing the scanning cost and improving the scanning efficiency of duplicate files.

[0022] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating a duplicate file scanning method in one embodiment of this application; Figure 2 yes Figure 1 A flowchart illustrating a specific implementation method of step 101; Figure 3 This is a flowchart illustrating a duplicate file scanning method in another embodiment of this application; Figure 4 yes Figure 1 A flowchart illustrating a specific implementation method for step 103; Figure 5 This is a flowchart illustrating a duplicate file scanning method in another embodiment of this application; Figure 6 This is a schematic diagram of the structure of a duplicate document scanning device in one embodiment of this application; Figure 7 This is a schematic diagram of the device structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0024] The invention will now be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are described merely to enable those skilled in the art to better understand and thus implement the invention, and are not intended to imply any limitation on the scope of the invention.

[0025] As used herein, the term "comprising" and its variations are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "at least partially based on". The terms "one embodiment" and "an embodiment" are to be interpreted as "at least one embodiment". The term "another embodiment" is to be interpreted as "at least one other embodiment".

[0026] In related technologies, duplicate file scanning mainly relies on full hash calculation. This process requires traversing the file system, reading file content one by one, and calculating a complete hash value for each file, such as MD5 or SHA, to determine file duplication. However, with the increase in storage capacity and the surge in the number of files, problems arise such as excessively long scanning time and high CPU / disk I / O resource consumption. Especially in scenarios involving large file processing and frequent scanning, resources are repeatedly consumed, resulting in resource waste. This is specifically reflected in the following aspects: Full scan is inefficient: Traditional tools need to traverse the entire file system, open files one by one to read content, and scan deep directories or large files (such as videos, compressed files, etc.) can take several hours. Disk I / O usage often exceeds 90%, causing system lag. Hash calculation resources are wasted: the complete hash value is calculated for all files, ignoring metadata features such as file size and modification time. A large number of files of different sizes are calculated indiscriminately, and the peak CPU utilization often exceeds 80%. Furthermore, the hash value of the same file is calculated repeatedly (such as scanning the same directory multiple times). Lack of dynamic update capability: After the file system changes (such as adding / deleting / modifying files), a full scan is required again, and the scan results cannot be updated incrementally. This results in a lot of repetitive work in scenarios with frequent file operations (such as download directories and project folders). The large file processing strategy is crude: it directly calculates the complete hash of GB-level large files without distinguishing the differences in file content. This results in large files with only partial modifications (such as video clips) still needing to be fully calculated, wasting more than 90% of computing power.

[0027] To address this problem, this embodiment provides a method for scanning duplicate files, such as... Figure 1 As shown, it includes the following steps: 101. Obtain file metadata that meets the query criteria through the file attribute database.

[0028] In this embodiment, file attribute data is independent of the actual file content. It is a structured data set used to record file description information. Its core function is to record descriptive information for all files, without storing the actual file content, such as document text or video footage. Generally, the file attribute database is independent of the file content and can be accessed without opening or reading the file itself, resulting in extremely fast reading speeds. Because the file attribute database records data according to a fixed field format, it does not change with the file content. File metadata is the structured descriptive information recorded by the file attribute database for each file. Similar to the file attribute database, file metadata also does not store the actual file content and can be accessed without opening or reading the file itself, such as size, creation time, modification time, and MFT record number.

[0029] The core fields of a specific file attribute database include at least the following: Identification fields, such as file path, filename, and unique file ID, used to uniquely locate files; size fields, such as actual file size in bytes and disk category size, which are key fields for determining file duplication; time fields, such as creation time, last modification time, and last access time, used to differentiate file purpose and control access scope; type and permission fields, such as file format, file attributes, and access permissions, used to differentiate file purpose and control access scope; and storage fields, such as the file's storage location on disk, used to help the system quickly locate the file's physical storage address.

[0030] In this embodiment, the query conditions are the conditions set for filtering file metadata. Specifically, the query conditions can be range-based, specifying the storage range of the file, such as partition / path, directory level, etc. Query conditions can also be attribute-based, filtering based on file attributes, such as size, time, type, etc. Query conditions can also be combined, using multiple types together, such as directory level and size, path and time. Specifically, the metadata field identifier to be extracted can be determined based on the query conditions. Since the file attribute database is maintained by the system and accessed through the system interface, there is no need to directly manipulate the database file. The system interface can be directly called to perform field matching in the file attribute database based on the metadata field identifier. The preset metadata fields are then extracted from the matched database records. After structural standardization processing, the file metadata that meets the query conditions is obtained.

[0031] 102. Based on the file metadata that meets the query conditions, establish a structured index library.

[0032] In this embodiment, each record in the structured index is stored in a formatted combination of defined fields, and all records are stored in an ordered tree structure. Specifically, based on at least one pre-determined search field, the file metadata that meets the query conditions is standardized. A primary index is set up with the first defined search field as the core and associated with the second search field. An auxiliary index is set up with an inverted index built for the third defined search field. The standardized file metadata is imported in batches into the storage carrier of the primary index according to the correspondence of the primary index. All records in the primary index are traversed and categorized according to the second defined field, so that records with the same field attributes are associated and constructed according to the auxiliary index. The entire process revolves around the determined search fields. Through precise positioning of the primary index and rapid grouping of the auxiliary index, the file metadata is transformed from a scattered state into a highly usable structured database, providing underlying support for subsequent operations such as duplicate file identification and file retrieval, significantly reducing invalid traversal and lowering resource consumption.

[0033] Specifically, in the process of batch importing file metadata into the main index, taking a memory scenario as an example, all file metadata can be traversed, using the file path as the first set search field, and the file size, timestamp, and optional fields as the second index fields, storing the file metadata one by one into the hash table of the main index. Taking a database scenario as an example, a batch import statement can be executed to import all file metadata into the data table at once; the database automatically maintains the uniqueness of the primary key and rejects duplicate paths. Taking a custom scenario as an example, according to a set format, the index fields of each file metadata record can be written sequentially into the main index storage area, recording the storage address of each record.

[0034] Specifically, in the process of building associations based on secondary indexes, taking a memory scenario as an example, all records in the primary index can be traversed, and the file size and path list can be extracted as third-defined fields. If the third-defined field exists in the secondary index, the path is added to the list; otherwise, a new value pair is created for the file size and path list. Taking a database scenario as an example, a query index can be created for the third-defined field of the data table. The database will automatically build the mapping for the third-defined field, eliminating the need to manually maintain the path list; queries can directly filter through the index. Taking a custom scenario as an example, each record in the primary index can be traversed, and the file path storage addresses with the same field attribute as the third-defined field can be organized into a list and written to the secondary index storage area, forming a list association of the third-defined field.

[0035] 103. Perform multi-level progressive checks on system files based on the structured index library to identify duplicate files in the system.

[0036] In this embodiment, multi-level progressive testing can transform a structured index library into logical pairs of system files with different granularities, progressively narrowing down the scope to identify duplicate files in the system. Specifically, multiple testing levels with progressively smaller granularities can be set using the index structure of the structured index library, along with field conditions applicable to each testing level. For each testing level, the field conditions are used to filter duplicate files in the system.

[0037] The first verification level is the most granular. This level leverages basic retrieval fields from the structured index's metadata to quickly eliminate files that are impossible to duplicate, significantly reducing subsequent comparisons. For example, files are first grouped by size, directly separating files of different sizes into different groups. Only files within the same group are likely to be duplicates, and these are considered duplicates identified by the first verification level. Since files filtered through the first verification level may still be non-duplicate, the second verification level typically uses a finer granularity. This level further narrows the scope using feature retrieval fields from the structured index's metadata, focusing on high-probability duplicate file clusters. For example, partial hash calculations are performed on files of the same size identified by the first verification level, and files with identical hash values ​​are considered duplicates identified by the second verification level. The third verification level, building upon the high-probability duplicate files identified by the second verification level, uses retrieval fields with reserved complete content features in the structured index to eliminate false positives. On one hand, the complete hash value of the file can be calculated and compared with the corresponding complete hash value of the file stored in the structured index. If the hash values ​​match, the file is determined to be a duplicate. On the other hand, if the structured search library stores deep features such as binary fragment indexes and text summaries of the file, files in special scenarios can be specifically verified. For example, for files after format conversion, some hash values ​​may differ, but if the complete content hash is consistent or the text summary matches completely, duplicate files can be further confirmed. Finally, a list of duplicate files is generated by combining different verification levels. This list clearly lists the path, size, and number of times duplicate files are repeated, and is associated with file priority tags in the structured index to provide a basis for subsequent cleanup. For example, system files are marked as undeletable, and ordinary documents are marked as cleanable.

[0038] In practical applications, the aforementioned duplicate file scanning method, with its efficient and accurate identification capabilities, can adapt to different file systems and storage sizes on various devices. Specifically, it can be applied to devices used for file organization, including personal consumer devices such as mobile phones and personal computers, as well as enterprise / office scenarios such as enterprise file servers and backup servers. The duplicate file scanning method provided in this application, compared with the existing technology that relies on full hash calculation to achieve duplicate file scanning, obtains file metadata that meets the query conditions through a file attribute database. The file attribute data is independent of the actual file content and is a structured data set used to record file description information. The file metadata is the structured description information recorded by the file attribute database for each file. Based on the file metadata that meets the query conditions, a structured index library is established. Each record in the structured index library is formatted and stored with a set combination of fields, and all records are stored in an ordered tree structure. The system files are subjected to multi-level progressive verification based on the structured index library to identify duplicate files in the system. The entire process establishes a structured index through metadata independent of the actual file content, ensuring that the index library records are accurately associated with the system files, improving the system file reading efficiency. Subsequent duplicate file identification uses multi-level verification coverage to adapt to different scenarios. While ensuring the accuracy of duplicate file identification results, only a small number of system files need to be fully verified, which greatly reduces the scanning cost of duplicate files and improves the scanning efficiency.

[0039] Understandably, the file attribute database already stores the metadata of all files in a structured format. This eliminates the need to open each file individually to read its content; instead, it allows for duplicate file scanning by reading only a small amount of structured information, significantly reducing resource consumption. Specifically, for example... Figure 2 As shown, step 101 includes the following steps: 201. Open the disk volume in the file attribute database using system functions to obtain a volume handle with read permissions.

[0040] 202. Call the device control function to initiate a query request to the file attribute database, so as to pass the volume handle with read permission and the pre-configured file query parameters to the file attribute database through the query request.

[0041] 203. Batch parse the continuous structure output by the file attribute database in the buffer to obtain file metadata that meets the query conditions.

[0042] In this embodiment, the volume handle is used to access the file attribute database, which is equivalent to establishing a direct connection with the target disk's file attribute database. All subsequent queries are initiated through this handle, avoiding the overhead of repeatedly establishing connections. By specifying the disk volume path through system functions, read permissions can be requested, and the volume handle can be obtained.

[0043] It should be noted that the system will verify permissions. Only volume handles with authorized access will be returned, while disks with insufficient permissions will be rejected, ensuring operational security.

[0044] In this embodiment, the device control function is a low-level communication channel provided by the system, directly connecting to the file attribute database kernel layer. Compared to upper-layer interfaces, it is more efficient and supports more complex sets of query parameters. The device control function can package the acquired volume handle and pre-set query parameters into a query request and send it to the file attribute data. Here, the query parameters must be encapsulated according to the system's specified format to ensure the database can parse them; for example, size conditions are converted to bytes, and time conditions are converted to timestamps.

[0045] Understandably, after receiving a query request, the file attribute database outputs a continuous stream of structured data to the system buffer. Each structure corresponds to a matching file metadata record, which includes fields such as path, size, and timestamp. The program then parses these structures in batches to extract the required file metadata. Here, the structures are in the database's native data format, requiring no intermediate format conversion, resulting in extremely high batch parsing efficiency.

[0046] Specifically, during the batch parsing of continuous structures output from the file attribute database in the buffer, file metadata can be extracted according to the field offsets defined in the structure. For example, bytes 1-4 of the structure represent the file size, and bytes 5-12 represent the timestamp. This ensures accurate field correspondence and filters invalid structures, such as abnormal records with empty fields. Correspondingly, starting from the beginning of the file attribute data output in the buffer, each structure output from the buffer is traversed and parsed until all structures output from the buffer are completed, yielding the file metadata. Pre-defined query conditions are then used to filter the parsed file metadata, obtaining file metadata that meets the query criteria.

[0047] In practical applications, since the core data of structured indexes comes from file system metadata, changes to the file system directly alter this fundamental information. For example, if a new file is added to the system but the structured index is not updated, it will miss the file, leading to misidentification of duplicate files. To ensure consistency between the structured index and the actual file system, further... Figure 3 As shown, after step 102, the method further includes the following steps: 301. When the file system changes, the structured index is logically updated using the metadata change records captured by the file attribute database.

[0048] 302. If the file system does not support setting change records, the modified attributes of the current file are used as a feature combination reflecting the file status, and the structured index is logically updated when it is determined that the metadata has changed.

[0049] In this embodiment, the file attribute database records all changes to the file system in real time. These change records need to be captured through system mechanisms. Each captured change record must include the operation type (add / delete / modify), the file's unique identifier (path or inode number), and the changed metadata (such as the modified size, timestamp, and complete metadata of the newly added file). One possible implementation is to capture system file change records through system event listening. This triggers a registered file system change notification; when a file is added, deleted, or modified, the system actively pushes the change event and associated metadata. Another possible implementation is to capture system file change records using incremental scanning. This method periodically compares the file attribute data change logs and extracts the change records added since the last synchronization, avoiding a full scan.

[0050] In practical applications, for captured change records, the system file change type is analyzed, and the locations of content requiring adjustment are marked in the structured index based on the change type. For the change type of newly added files, their complete metadata is parsed to determine if they meet the query conditions of the structured index; if so, they are marked as needing index insertion. For the change type of deleted files, the corresponding record in the main index is located by file path or unique ID, and marked as needing index deletion. For the change type of modified files, the focus is on analyzing whether it affects index fields; if core index fields change, it is marked as needing index updates. Further, logical updates are performed at the marked locations in the structured index according to the change type. For the change type of newly added files, the key-value pairs of the newly added file's metadata are inserted into the main index, and the file path is added to the secondary index. For the change type of deleted files, the corresponding record in the main index is located and deleted by path, and the file path list is found in the secondary index, and its path is removed. For the change type of modified files, if only the path has changed, the old path information is deleted from the main index, and the association between the new path and metadata is inserted; the secondary index does not need adjustment. If the size changes, the path is grouped and moved in the secondary index, and then the changed fields are updated in the primary index. If only the timestamp changes, only the last time field that needs to be changed in the primary index is updated; the secondary index does not need to be adjusted.

[0051] It should be noted that due to limitations in file system type, storage device characteristics, system configuration, or underlying technology, file systems may not support change log settings in certain scenarios, and therefore cannot natively provide real-time change logs or event notifications. In such cases, it is necessary to use the modified attributes of the current file as a combination of features reflecting the file's state, and logically update the structured index when it is determined that metadata has changed. Specifically, periodic incremental verification can replace real-time change logging. The verification frequency and scope are set according to the scenario. By using the file attribute database, the actual combination of features of the current file is obtained, and each feature in the actual feature combination is compared field by field. If there are inconsistencies, it is determined that an update is required.

[0052] In practical applications, considering the sequential progression of multi-layered progressive verification, where the output of the previous layer serves as the input for the next, higher-level system files consume fewer resources, while lower-level system files consume more. Therefore, a multi-layered progressive verification system is pre-built, which includes at least file byte verification, partial hash verification, and full verification based on the partial hash verification results. Specifically, such as... Figure 4 As shown, step 103 includes the following steps: 401. Extract statistical characteristics of system files based on the number of bytes according to the structured index library, and use system files with the same number of bytes as the statistical characteristics as the first-level verification files.

[0053] 402. In the first-level verification file, the sampled content of each system file is read one by one, and the system file with a consistent hash value of the sampled content is used as the second-level verification file.

[0054] 403. In the second-level verification file, read the complete content of each system file one by one, and obtain the system files whose complete content has a consistent hash value as duplicate files in the system.

[0055] Specifically, in the first-level verification process, the core premise for duplicate files is that their content is completely identical. Identical content necessarily means the same number of bytes. Conversely, system files with different byte counts, regardless of other attributes, cannot be duplicate files and can be directly eliminated without any further verification. For example, the structured index can be used to iterate through all path lists corresponding to different byte counts, filtering out groups with path list lengths ≥ 2 bytes. All system files in these groups are then extracted as first-level verification files. This process can determine whether a file is a system file using file attribute fields in the structured index, i.e., files marked as system-level and undeletable.

[0056] Specifically, in the second-level verification process, based on the paths of the first-level verification files, the file system is accessed one by one. According to the set sampling rules, system files are opened and sampled fragments are read. Hash values ​​are calculated for each sampled fragment. An efficient and highly discriminative algorithm is used to record the path-sampled hash value correspondence for each file, and this is associated with file identifiers in the structured index. Only files with completely identical sampled hash values ​​are retained in the first-level verification files as the second-level verification files. It can be understood that the second verification process mainly narrows down the candidate set to a very small number of high-probability duplicate files through sampling hashing, significantly reducing the pressure on the third verification.

[0057] The essence of duplicate files is that their byte-level content is completely identical. Relying solely on sampling hashing may lead to false positives due to coincidental similarities in file segments. Full content hashing, by calculating the hash value of each byte of the file, can uniquely identify the file content and is the ultimate standard for determining duplicates. Specifically, in the third-level verification process, based on the path of the second-level verification file, the file system is accessed one by one, all byte content is read, a hash algorithm with stronger collision prevention is selected, the hash value of the complete content is calculated, the path-full hash value correspondence of each file is recorded, and the metadata in the structured index is linked. In the second-level verification file, only files with completely identical full hash values ​​are retained as duplicate files in the system.

[0058] In practical applications, hash calculations consume CPU resources, and file reading requires disk I / O. Hash caching mechanisms can directly reuse historical results, reducing redundant calculations and file reads. Furthermore, such as... Figure 5 As shown, after step 103, the method further includes the following steps: 501. Associate and bind the hash value obtained by multi-level progressive verification with the file path to form a mapping relationship.

[0059] 502. In response to the hash value call of the system file to be queried, locate the hash value of the system file to be queried based on the file path in the mapping relationship.

[0060] In this embodiment, the hashable caching mechanism mainly targets the sampled hash value and the full hash value calculated in the multi-layer progressive verification, which are bound to the absolute path of the file and the file identifier in the structured index library, respectively, to form a structured mapping table.

[0061] In scenarios where repeated verification is performed during periodic index updates, the same file needs to be verified multiple times. If the sampled / full hash value is already stored in the cache, there is no need to read the file again and recalculate; the cached result can be directly called, which helps improve efficiency.

[0062] In the scenario of verifying duplicate files, if A and B are duplicate files, during the verification process within the duplicate file group, the full hash of A is calculated and stored in the cache. When verifying B, the hash value of A in the cache can be directly compared, without having to calculate the hash of B again, thus reducing the amount of computation.

[0063] Furthermore, as Figure 1-5 To specifically implement the method, this application provides a duplicate document scanning device, such as... Figure 6 As shown, the device includes: an acquisition unit 611, an establishment unit 62, and an identification unit 63.

[0064] The acquisition unit 61 is used to acquire file metadata that meets the query conditions through the file attribute database. The file attribute data is unique to the actual content of the file and is a structured data set used to record file description information. The file metadata is the structured description information recorded by the file attribute database for each file. Establishment unit 62 is used to establish a structured index library based on the file metadata that meets the query conditions. Each record in the structured index library is stored in a formatted combination of set fields, and all records are stored in an ordered manner through a tree structure. The identification unit 63 is used to perform multi-level progressive verification on system files based on the structured index library in order to identify duplicate files in the system.

[0065] The duplicate file scanning device provided in this invention, compared with the existing technology that relies on full hash calculation to achieve duplicate file scanning, obtains file metadata that meets the query conditions through a file attribute database. The file attribute data is independent of the actual file content and is a structured data set used to record file description information. The file metadata is the structured description information recorded by the file attribute database for each file. A structured index is established based on the file metadata that meets the query conditions. Each record in the structured index is formatted and stored with a set combination of fields, and all records are stored in an ordered tree structure. Multi-level progressive checks are performed on system files based on the structured index to identify duplicate files in the system. The entire process establishes a structured index using metadata independent of the actual file content, ensuring accurate association between index records and system files, improving system file reading efficiency. Subsequent duplicate file identification uses multi-level verification coverage to adapt to different scenarios. While ensuring the accuracy of duplicate file identification results, only a small portion of system files need to be fully verified, greatly reducing the scanning cost and improving the scanning efficiency of duplicate files.

[0066] In specific application scenarios, the acquisition unit includes: The acquisition module is used to open the disk volume in the file attribute database through system functions in order to obtain a volume handle with read permissions; The initiation module is used to call the device control function to initiate a query request to the file attribute database, so as to pass the volume handle with read permission and the pre-configured file query parameters to the file attribute database through the query request; The parsing module is used to parse the continuous structures output by the file attribute database in the buffer in batches to obtain file metadata that meets the query conditions. Each structure corresponds to a file metadata record.

[0067] In specific application scenarios, the parsing module is specifically used for: Starting from the beginning position of the file attribute data output in the buffer, traverse and parse each structure output in the buffer until all structures output in the buffer are traversed to obtain the file metadata; The parsed file metadata is filtered using pre-defined query criteria to obtain file metadata that meets the query criteria.

[0068] In specific application scenarios, the matching unit includes: The update unit is used to logically update the structured index library by capturing metadata change records from the file attribute database when the file system changes after the structured index library is established based on the file metadata that meets the query conditions. The update unit is further configured to, if the file system does not support the setting of change records, use the modified attributes of the current file as a feature combination reflecting the file status, and logically update the structured index library when it is determined that the metadata has changed.

[0069] In specific application scenarios, the device further includes: The construction unit is used to pre-construct a multi-layer progressive verification before performing multi-layer progressive verification on system files according to the structured index library to identify duplicate files in the system. The multi-layer progressive verification includes at least file byte verification, partial hash verification, and full verification based on the partial hash verification results.

[0070] In specific application scenarios, the identification unit is specifically used for: Based on the structured index, statistical characteristics of system files based on the number of bytes are extracted, and system files with the same number of bytes based on the statistical characteristics are used as the first-level verification files; In the first-level verification file, the sampled content of each system file is read one by one, and the system file with a consistent hash value of the sampled content is used as the second-level verification file; In the second-level verification file, the complete content of each system file is read one by one, and system files with consistent hash values ​​in their complete content are identified as duplicate files in the system.

[0071] In specific application scenarios, the device further includes: The binding unit is used to associate and bind the hash value calculated by the multi-level progressive verification with the file path after the system files are identified by performing multi-level progressive verification according to the structured index library to identify duplicate files in the system, thereby forming a mapping relationship. The locating unit is used to locate the hash value of the system file to be queried in response to the hash value call of the system file to be queried, based on the file path in the mapping relationship.

[0072] It should be noted that other corresponding descriptions of the functional units involved in the duplicate document scanning device provided in this embodiment can be found in [reference needed]. Figures 1-5 The corresponding descriptions in [the document] will not be repeated here.

[0073] Based on the above, Figures 1-5 Accordingly, this application embodiment also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method. Figures 1-5 The method for scanning duplicate files is shown.

[0074] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0075] Based on the above, Figures 1-5 The method shown, and Figure 6 To achieve the above objectives, the present application also provides a physical device for repeated document scanning, which can be a computer, smartphone, tablet, smartwatch, server, or network device, etc. This physical device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described... Figures 1-5 The method for scanning duplicate files is shown.

[0076] Optionally, the physical device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0077] In an exemplary embodiment, see Figure 7 The aforementioned physical device includes a communication bus, a processor, a memory, and a communication interface. It may also include input / output interfaces and a display device. The various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the repetitive file scanning method described in the above embodiments.

[0078] Those skilled in the art will understand that the physical device structure for repeated document scanning provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0079] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device for the aforementioned repetitive file scanning, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. Compared with existing methods, by applying the technical solution of this application, this application establishes a structured index using metadata independent of the actual content of the file, ensuring accurate association between the index records and system files, improving the efficiency of system file reading. Subsequent duplicate file identification uses multi-layered verification coverage to adapt to different scenarios. While ensuring the accuracy of duplicate file identification results, only a small portion of system files need to be fully verified, greatly reducing the scanning cost of duplicate files and improving the scanning efficiency.

[0081] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0082] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for scanning duplicate files, characterized in that, include: The file metadata that meets the query conditions is obtained through the file attribute database. The file attribute data is unique to the actual content of the file and is a structured data set used to record file description information. The file metadata is the structured description information recorded by the file attribute database for each file. Based on the file metadata that meets the query conditions, a structured index library is established. Each record in the structured index library is stored in a formatted way with a set combination of fields, and all records are stored in an ordered manner through a tree structure. The system files are subjected to multi-level progressive verification based on the structured index library to identify duplicate files in the system.

2. The method according to claim 1, characterized in that, The step of obtaining file metadata that meets the query conditions through the file attribute database includes: Open the disk volume in the file attribute database using system functions to obtain a volume handle with read permissions; The device control function is invoked to initiate a query request to the file attribute database, so as to pass the volume handle with read permission and the pre-configured file query parameters to the file attribute database through the query request; The continuous structures output from the file attribute database in the buffer are parsed in batches to obtain file metadata that meets the query conditions. Each structure corresponds to a file metadata record.

3. The method according to claim 2, characterized in that, The batch parsing of the continuous structure output from the file attribute database in the buffer to obtain file metadata that meets the query conditions includes: Starting from the beginning position of the file attribute data output in the buffer, traverse and parse each structure output in the buffer until all structures output in the buffer are traversed to obtain the file metadata; The parsed file metadata is filtered using pre-defined query criteria to obtain file metadata that meets the query criteria.

4. The method according to claim 1, characterized in that, After the method establishes a structured index based on the file metadata that meets the query conditions, the method further includes: When the file system changes, the structured index is logically updated using the metadata change records captured by the file attribute database. If the file system does not support setting change records, the modified attributes of the current file are used as a combination of features reflecting the file status, and the structured index is logically updated when it is determined that the metadata has changed.

5. The method according to any one of claims 1-4, characterized in that, Before performing multi-level progressive checks on system files based on the structured index library to identify duplicate files in the system, the method further includes: A multi-layered progressive verification is pre-constructed, which includes at least file byte verification, partial hash verification, and full verification based on the partial hash verification results.

6. The method according to claim 5, characterized in that, The step of performing multi-level progressive checks on system files based on the structured index library to identify duplicate files in the system includes: Based on the structured index, statistical characteristics of system files based on the number of bytes are extracted, and system files with the same number of bytes based on the statistical characteristics are used as the first-level verification files; In the first-level verification file, the sampled content of each system file is read one by one, and the system file with a consistent hash value of the sampled content is used as the second-level verification file; In the second-level verification file, the complete content of each system file is read one by one, and system files with consistent hash values ​​in their complete content are identified as duplicate files in the system.

7. The method according to claim 6, characterized in that, After performing multi-level progressive verification on system files based on the structured index library to identify duplicate files in the system, the process includes: The hash value obtained by multi-level progressive verification is associated and bound with the file path to form a mapping relationship; In response to a call to the hash value of a system file to be queried, the hash value of the system file to be queried is located based on the file path in the mapping relationship.

8. A duplicate document scanning device, characterized in that, include: The acquisition unit is used to acquire file metadata that meets the query conditions through the file attribute database. The file attribute data is unique to the actual content of the file and is a structured data set used to record file description information. The file metadata is the structured description information recorded by the file attribute database for each file. The establishment unit is used to establish a structured index library based on the file metadata that meets the query conditions. Each record in the structured index library is stored in a formatted combination of set fields, and all records are stored in an ordered manner through a tree structure. The identification unit is used to perform multi-level progressive checks on system files based on the structured index library to identify duplicate files in the system.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the duplicate file scanning method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the duplicate file scanning method according to any one of claims 1 to 7.

Images