File checking method, system, device and equipment and storage medium

Abstract

The invention provides a file checking method, system, device and equipment and a storage medium, and the method comprises the steps that in response to an execution instruction for a file checking task, a target file main data table is determined, file information of a target file is stored in the target file main data table, and the file information of the target file is stored in the target file main data table; the file information is obtained from a plurality of file data tables in advance; and when it is determined that the files which have the same identification information and different file access addresses in the target file main data table exist, marking the files as duplicate files corresponding to the target files. Visibly, the file information in the multiple file data tables can be integrated in the target file main data in advance, when the file uniqueness is checked, only the file checking task needs to be executed for the file main data table, file checking files do not need to be configured for the multiple file data tables, and therefore consumption of computing resources is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a file checking method, system, device, equipment and storage medium. Background Art

[0002] File uniqueness check refers to checking files in a computer system to ensure that each file is unique in the entire system, that is, there are no duplicate files with the same content. File uniqueness check can ensure that files are not duplicated or tampered with during storage or transmission.

[0003] Currently, when checking multiple files in the system, corresponding file checking tasks are usually configured for multiple data tables containing file information for inspection. Obviously, this method of configuring multiple file checking tasks for inspection requires scanning and querying a large number of data tables, which consumes more system resources. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present disclosure provide a file checking method, system, device, equipment and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a file checking method, the method comprising:

[0006] In response to an execution instruction for a file checking task, a target file master data table is determined; wherein the target file master data table stores file information of the target file, the file information is obtained from a plurality of file data tables, the file information includes a file access address, and the file information is used to determine identification information of the target file;

[0007] If it is determined that there is a file in the target file master data table with the same identification information as the target file and a different file access address, the file is determined as a duplicate file corresponding to the target file.

[0008] In an optional implementation method, the method further includes:

[0009] In response to an execution instruction for a data integration task, scanning the plurality of file data tables; wherein the data integration task carries file configuration information, and the file configuration information is used to determine a data table containing file information;

[0010] Matching the scanned metadata information of the plurality of file data tables with the file configuration information;

[0011] If it is determined that the metadata information of the target file data table in the multiple file data tables successfully matches the file configuration information, the metadata information of the target file data table is stored in the file master data table.

[0012] In an optional implementation method, the file configuration information includes a file name and file access address information, and if it is determined that the metadata information of the target file data table in the multiple file data tables successfully matches the file configuration information, the metadata information of the target file data table is stored in the file master data table, including:

[0013] If it is determined that the metadata information of the target file data table among the multiple file data tables successfully matches at least the file name and the file access address information in the file configuration information, the metadata information of the target file data table is stored in the file master data table based on the format requirements of the file master data table.

[0014] In an optional implementation method, the file information further includes a storage server type, and before determining whether there is a file in the target file master data table that has the same identification information as the target file and a different file access address from the target file, the method further includes:

[0015] Based on the file access address and the storage server type, identification information of the target file is determined.

[0016] In a second aspect, the present disclosure provides a file inspection system, the system comprising a task scheduling module and a data quality module;

[0017] The task scheduling module is used to send an execution instruction for the file inspection task to the data quality module when the start execution time of the file inspection task is monitored;

[0018] The data quality module is used to determine the target file master data table when receiving the execution instruction for the file inspection task; wherein the target file master data table stores file information of the target file, the file information is obtained from multiple file data tables, the file information includes a file access address, and the file information is used to determine the identification information of the target file;

[0019] The data quality module is also used to determine whether there is a file in the target file master data table that has the same identification information as the target file and a different file access address from the target file, and when it is determined that there is a file in the target file master data table that has the same identification information as the target file and a different file access address, the file is determined as a duplicate file corresponding to the target file.

[0020] In an optional implementation method, the system further includes a data integration module;

[0021] The task scheduling module is further used to send an execution instruction for the data integration task to the data integration module when the start execution time of the data integration task is monitored;

[0022] The data integration module is used to scan the multiple file data tables when receiving an execution instruction for the data integration task; wherein the data integration task carries file configuration information, and the file configuration information is used to determine the data table containing the file information;

[0023] The data integration module is also used to match the metadata information of the scanned multiple file data tables with the file configuration information, and when it is determined that the metadata information of the target file data table in the multiple file data tables successfully matches the file configuration information, the metadata information of the target file data table is stored in the file master data table.

[0024] In a third aspect, the present disclosure provides a document inspection device, the device comprising:

[0025] A first determination module is used to determine a target file master data table in response to an execution instruction for a file checking task; wherein the target file master data table stores file information of the target file, the file information is obtained from a plurality of file data tables, the file information includes a file access address, and the file information is used to determine identification information of the target file;

[0026] The marking module is used to mark the file as a duplicate file corresponding to the target file if it is determined that there is a file in the target file master data table that has the same identification information as the target file and a different file access address.

[0027] In a fourth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the above method.

[0028] In a fifth aspect, the present disclosure provides a file inspection device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0029] In a sixth aspect, the present disclosure provides a computer program product, wherein the computer program product comprises a computer program / instructions, and the computer program / instructions implement the above method when executed by a processor.

[0030] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has at least the following advantages:

[0031] The disclosed embodiments provide a file checking method, system, device, equipment and storage medium. Specifically, the method includes: in response to an execution instruction for a file checking task, determining a target file master data table, wherein the target file master data table stores file information of the target file, and the file information is obtained from a plurality of file data tables pre-configured; determining whether there is a file in the target file master data table that has the same identification information as the target file and a different access address from the target file, and when determining that there is a file in the target file master data table that has the same identification information as the target file and a different file access address, marking the file as a duplicate file corresponding to the target file. It can be seen that the disclosed embodiments can pre-integrate the file information in a plurality of file data tables into the target file master data, and when performing a file uniqueness check, it is only necessary to perform the file checking task for the file master data table, without configuring the file checking file for a plurality of file data tables, thereby reducing the consumption of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 A schematic diagram of the structure of a file inspection system provided by an embodiment of the present disclosure;

[0035] Figure 2 A flowchart of a file checking method provided by an embodiment of the present disclosure;

[0036] Figure 3 A schematic diagram of the structure of a document inspection device provided by an embodiment of the present disclosure;

[0037] Figure 4 A schematic diagram of the structure of a file inspection device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0039] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0040] File uniqueness check refers to checking files in a computer system to ensure that each file is unique in the entire system, that is, there are no duplicate files with the same content. File uniqueness check can ensure that files are not duplicated or tampered with during storage or transmission.

[0041] Currently, when checking multiple files in a system, file checking tasks are usually configured for all data tables containing file information for checking. Obviously, this method of configuring a large number of checking tasks consumes more computing resources.

[0042] To this end, the embodiment of the present disclosure provides a file checking method, specifically, in response to an execution instruction for a file checking task, determining a target file master data table, wherein the target file master data table stores file information of the target file, and the file information is obtained from a plurality of file data tables in advance; determining whether there is a file in the target file master data table that has the same identification information as the target file and a different access address from the target file, and when determining that there is a file in the target file master data table that has the same identification information as the target file and a different file access address, marking the file as a duplicate file corresponding to the target file. It can be seen that the embodiment of the present disclosure can integrate the file information in a plurality of file data tables in advance in the target file master data, and when performing a file uniqueness check, it is only necessary to execute the file checking task for the file master data table, without configuring corresponding checking tasks for a plurality of file data tables, thereby reducing the consumption of computing resources.

[0043] Based on this, the present disclosure provides a file inspection system. Figure 1 , is a structural diagram of a file inspection system provided in an embodiment of the present disclosure, wherein the file inspection system 100 includes a task scheduling module 101, a data integration module 102 and a data quality module 103.

[0044] The data integration module is used to receive table item information input by the user and create a file master data table based on the table item information. Exemplarily, the file master data table created by the user may include table item information such as table identifier, English name of the source table, file name, storage server type, file access address, file hash value, creation time, modification time, remarks, modification user name, etc.

[0045] The data integration module is also used to receive a data integration task configured by a user for the file master data table, wherein the data integration task carries file configuration information, which can be used to identify which data tables contain file information for file uniqueness check.

[0046] The file configuration information may include file name, file access address, storage server type and other information.

[0047] In an optional implementation, the data integration module may also support users to configure task modes for data integration tasks. Specifically, after receiving the data integration task configured by the user for the file master data table, the module may also receive task modes configured by the user for the data integration task, wherein the task modes may include full data scanning, data append scanning and other modes.

[0048] Specifically, full data scanning refers to the process of completely scanning and processing the entire data table. During the full data scanning process, the data integration module will traverse all data rows in the data table and process them regardless of whether the data has changed. Data append scanning refers to processing only the data partitions that have changed since the last scan without scanning the entire data table. For example, you can determine which data is new or modified by comparing the data's timestamp or incremental mark, and only process this part of the data.

[0049] In an optional implementation, the data integration module may also support users to configure the execution time of data integration tasks, such as configuring to execute a data integration task once a day to generate a file master data table, or to execute a data integration task once a week to generate a file master data table, etc.

[0050] In the disclosed embodiment, after receiving the data integration task configured by the user for the file master data table, as well as the task mode and execution time for the data integration task, the data integration module can also send the data integration task and the corresponding task mode and execution time to the task scheduling module, and the task scheduling module will monitor and schedule the data integration task.

[0051] The task scheduling module is used to send an execution instruction for the data integration task to the data integration module when the start execution time of the data integration task is monitored.

[0052] In the disclosed embodiment, the task scheduling module is used to schedule data integration tasks, file checking tasks, etc. configured by the user. Specifically, when the start execution time of the data integration task is monitored, an instruction to execute the data integration task is sent to the data integration module.

[0053] In the disclosed embodiment, the execution instruction is used to instruct the data integration module to start executing the data integration task. The data integration task carries file configuration information. The data integration module can determine the data table containing the file information in the database through the file configuration information.

[0054] The data integration module is further configured to scan multiple file data tables and match metadata information of the multiple file data tables with file configuration information when receiving an execution instruction for the data integration task, wherein the file configuration information is used to determine a data table containing file information.

[0055] In the embodiment of the present disclosure, in the process of scanning multiple file data tables, metadata information of the scanned file data tables is matched with file configuration information. The scanned file data tables may be part or all of the data tables in the database, and metadata refers to data used to describe data attributes, which may support functions such as indicating storage location, historical data, resource search, and file records.

[0056] The data integration module is further configured to store the data of the target file data table in the file master data table when it is determined that the metadata information of the target file data table in the multiple file data tables successfully matches the file configuration information.

[0057] In an optional implementation, since the file master data table is pre-created based on the table entry information input by the user, when it is determined that the metadata information of the target file data table successfully matches the file configuration information, the metadata information of the target file data table can also be stored in the file master data table according to the table entry information based on the table entry information of the file master data table.

[0058] For example, assuming that the file configuration information is configured with file names such as wjmc, file_name, sjzwm, when the metadata information of file data table A is scanned, the metadata information of table A is matched with the above file names. If any file name among wjmc, file_name, sjzwm exists in the metadata information of table A, the metadata information of table A is stored in the file master data table according to the format requirements (or table item information) of the file master data table.

[0059] It can be seen that the embodiment of the present disclosure can integrate the data of the data table containing file information into the file master data table through the data integration task, so that when the file uniqueness check task is subsequently executed, it is only necessary to query the data in the file master data table without querying the data in all tables, thereby reducing the resource consumption of the system.

[0060] The data quality module is used to receive the file inspection task configured by the user for the file master data table. The file inspection task may include data inspection scope information for the file master data table, such as full table query or partial data query for the file master data table, and the file inspection task may also include task scheduling cycle information, such as immediate execution, periodic execution, and single-time execution.

[0061] In the disclosed embodiment, after receiving the file inspection task configured for the file master data table and the scheduling cycle information configured for the file inspection task, the data quality module can also send the inspection task and the corresponding scheduling cycle to the task scheduling module, and the task scheduling module will monitor and schedule the file inspection task.

[0062] The task scheduling module is also used to send an execution instruction for the file checking task to the data quality module when the start execution time of the file checking task is monitored.

[0063] The data quality module is further configured to determine a target file master data table when receiving an execution instruction for the file inspection task, wherein the target file master data table stores file information of the target file, and the file information is obtained from multiple file data tables.

[0064] Then, determine whether there is a file in the target file master data table that has the same identification information as the target file but a different file access address, and if there is a file in the target file master data table that has the same identification information as the target file but a different file access address, mark the file as a duplicate file corresponding to the target file.

[0065] In an optional embodiment, the data quality module is also used to query the file master data table for data that meets the data inspection range information based on the data inspection range information in the file inspection task when receiving an execution instruction for the file inspection task, and store the queried data that meets the data inspection range information in the inspection result table.

[0066] Exemplarily, the field information in the inspection result table may include task instance ID, English name of the source table, file name, storage server type, file access address, whether it is a whitelist, whether it is a duplicate, creation time, and other information.

[0067] In the disclosed embodiment, each row of data stored in the inspection result table corresponds to a file. After obtaining the inspection result table, any row of data in the inspection result table can be scanned and the file identification information can be generated based on the file access address and storage server type corresponding to the row of data.

[0068] In practical applications, the MD5 value of the file can be obtained by performing hash calculation on the file access address and the storage server type as the identification information of the file.

[0069] It can be seen that the embodiment of the present disclosure can integrate the file information in multiple file data tables into the target file master data in advance. When performing a file uniqueness check, it is only necessary to perform the file check task on the file master data table, without configuring file check files for multiple file data tables, thereby reducing the consumption of computing resources.

[0070] Based on the above method embodiments, the present disclosure provides a file inspection method, such as Figure 2 , which is a flow chart of a file checking method provided by an embodiment of the present disclosure.

[0071] S201: In response to an execution instruction for a file checking task, determining a target file master data table.

[0072] The target file master data table stores file information of the target file, the file information is obtained from a plurality of file data tables, the file information includes a file access address, and the file information is used to determine identification information of the target file.

[0073] In the disclosed embodiment, the file information of the target file may include the English name of the source table, file name, storage server type, file access address, file hash value, creation time, modification time, remarks, modification user name and other information.

[0074] In the disclosed embodiment, the file information may be used to determine identification information of the target file. Specifically, the MD5 value of the target file may be generated as identification information according to the storage server type and file access address information in the file information.

[0075] In an optional implementation, the file information of the target file can also be obtained from multiple file data tables according to the data integration task. Specifically, when an execution instruction for the data integration task is received, multiple file data tables are scanned, and the metadata information of the scanned multiple file data tables is matched with the file configuration information.

[0076] The data integration task carries file configuration information, which can be used to identify which data tables contain file information for file uniqueness check. The file configuration information can include global information such as file name, file access address, storage server type, etc.

[0077] For example, assuming that storage server types such as fwqlx and wjcjfwq are configured in the file configuration information, when scanning multiple file data tables and matching the metadata information of the scanned file data tables with the file configuration information, if the metadata information of file data table A contains the fwqlx storage server type, it can be determined that the metadata information of file data table A matches the file configuration information successfully, and at this time, the data of file data table A is stored in the file master data table.

[0078] In an optional implementation, since the format requirements of the file master data table are determined based on the table item information input by the user, when it is determined that the metadata information of the target file data table successfully matches the file configuration information, the metadata information of the target file data table can also be stored in the file master data table based on the format requirements of the file master data table.

[0079] Exemplarily, assuming that the table entry information of the file master data table includes the English name of the source table, file name, storage server type, file access address, file MD5 code, whether it is on the whitelist, creation time, modification time, remarks, modifier user name and other information, the file configuration information includes the file name, and the metadata information of the target file data table A includes the file name, storage server type, creation time, file size, file format and other information. When determining that the metadata information of data table A, such as the file name, successfully matches the file configuration information, the file name, storage server type and creation time in the metadata information of data table A are stored in the file master data table according to the table entry information of the file master data table.

[0080] It should be noted that, since the file master data table does not include information such as file size and file format, there is no need to store the file size and file format in the metadata information of data table A in the file master data table.

[0081] S202: If it is determined that there is a file in the target file master data table that has the same identification information as the target file and a different file access address, the file is marked as a duplicate file corresponding to the target file.

[0082] In the disclosed embodiment, the target file master data table stores file information of multiple files. For any file among the multiple files, an MD5 value can be generated based on the file access address and the storage server type as identification information of the file.

[0083] In an optional implementation, after the identification information of the target file is generated based on the file access address and the storage server type, the identification information may also be stored in the inspection result data table.

[0084] In the disclosed embodiment, after corresponding identification information is generated for each file, the identification information and the file access address of each file are compared. If there are multiple files with the same identification information but different file access addresses, the multiple files can be marked as duplicate files.

[0085] In the disclosed embodiment, if the identification information of two files is the same but the file access addresses are different, it means that the contents or data of the two files are the same but are stored in two different places, and therefore they are duplicate files.

[0086] In actual applications, when an execution instruction for a file inspection task is received, a corresponding task instance ID can also be generated for the task. After executing the file inspection task and determining the duplicate files corresponding to the target file, the inspection results of the file inspection task can be checked based on the task instance ID, that is, to check whether duplicate files are detected in this inspection task.

[0087] In an optional implementation, a duplicate mark can also be displayed on the target file's details page, allowing users to manually mark duplicate files or cancel the mark, or set the target file as a whitelist file so that the target file will not be checked for file uniqueness in the future.

[0088] In the improved file uniqueness check method of the disclosed embodiment, in response to the execution instruction for the file check task, the target file master data table is determined, wherein the target file master data table stores the file information of the target file, and the file information is obtained from a plurality of file data tables pre-configured; it is determined whether there is a file in the target file master data table that has the same identification information as the target file and a different access address from the target file, and when it is determined that there is a file in the target file master data table that has the same identification information as the target file and a different file access address, the file is marked as a duplicate file corresponding to the target file. It can be seen that the disclosed embodiment can integrate the file information in a plurality of file data tables in the target file master data in advance, and when performing the file uniqueness check, it is only necessary to execute the file check task for the file master data table, without configuring the file check file for a plurality of file data tables, thereby reducing the consumption of computing resources.

[0089] Based on the above method embodiment, the present disclosure also provides a file inspection device, referring to Figure 3 , is a schematic diagram of the structure of a file inspection device provided by an embodiment of the present disclosure, the device comprising:

[0090] The first determination module 301 is used to determine a target file master data table in response to an execution instruction for a file checking task; wherein the target file master data table stores file information of the target file, the file information is obtained from a plurality of file data tables, the file information includes a file access address, and the file information is used to determine identification information of the target file;

[0091] The marking module 302 is used to mark the file as a duplicate file corresponding to the target file if it is determined that there is a file in the target file master data table that has the same identification information as the target file but a different file access address.

[0092] In an optional implementation, the device further includes:

[0093] A scanning module, configured to scan the plurality of file data tables in response to an execution instruction for a data integration task; wherein the data integration task carries file configuration information, and the file configuration information is used to determine a data table containing file information;

[0094] A matching module, used for matching the scanned metadata information of the plurality of file data tables with the file configuration information;

[0095] The storage module is used to store the metadata information of the target file data table in the file master data table if it is determined that the metadata information of the target file data table in the multiple file data tables successfully matches the file configuration information.

[0096] In an optional implementation manner, the file configuration information includes a file name and file access address information, and the storage module includes:

[0097] The storage submodule is used to store the metadata information of the target file data table in the file master data table if it is determined that the metadata information of the target file data table in the multiple file data tables successfully matches at least the file name and the file access address information in the file configuration information.

[0098] In an optional implementation manner, the file information further includes a storage server type, and the device further includes:

[0099] The second determination module is used to determine the identification information of the target file based on the file access address and the storage server type.

[0100] In the file checking device provided by the embodiment of the present disclosure, in response to the execution instruction for the file checking task, the target file master data table is determined, wherein the target file master data table stores the file information of the target file, and the file information is obtained from a plurality of file data tables pre-configured; it is determined whether there is a file in the target file master data table that has the same identification information as the target file and a different access address from the target file, and when it is determined that there is a file in the target file master data table that has the same identification information as the target file and a different file access address, the file is marked as a duplicate file corresponding to the target file. It can be seen that the embodiment of the present disclosure can integrate the file information in a plurality of file data tables in the target file master data in advance, and when performing the file uniqueness check, it is only necessary to execute the file checking task for the file master data table, without configuring the file checking file for a plurality of file data tables, thereby reducing the consumption of computing resources.

[0101] In addition to the above-mentioned method and apparatus, the embodiments of the present disclosure further provide a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device implements the file uniqueness checking method described in the embodiments of the present disclosure.

[0102] The embodiment of the present disclosure further provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the file checking method described in the embodiment of the present disclosure is implemented.

[0103] In addition, the present disclosure also provides a document inspection device, see Figure 4 As shown, it may include:

[0104] Processor 401, memory 402, input device 403 and output device 404. The number of processors 401 in the file inspection device can be one or more. Figure 4 In some embodiments of the present disclosure, the processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means, wherein: Figure 4 The example of connecting through bus is taken in the following.

[0105] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing of the document inspection device by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The input device 403 can be used to receive input digital or character information, and generate signal input related to user settings and function control of the document inspection device.

[0106] Specifically in this embodiment, the processor 401 will load the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 will run the applications stored in the memory 402, thereby realizing the various functions of the above-mentioned file inspection device.

[0107] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0108] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A file inspection method, characterized in that: The method comprises: In response to an execution instruction for a file checking task, a target file master data table is determined; wherein the target file master data table stores file information of the target file, the file information is obtained from a plurality of file data tables, the file information includes a file access address, and the file information is used to determine identification information of the target file; If it is determined that there is a file in the target file master data table with the same identification information as the target file and a different file access address, the file is marked as a duplicate file corresponding to the target file.

2. The method according to claim 1, characterized in that: The method further comprises: In response to an execution instruction for a data integration task, scanning the plurality of file data tables; wherein the data integration task carries file configuration information, and the file configuration information is used to determine a data table containing file information; Matching the scanned metadata information of the plurality of file data tables with the file configuration information; If it is determined that the metadata information of the target file data table in the multiple file data tables successfully matches the file configuration information, the metadata information of the target file data table is stored in the file master data table.

3. The method according to claim 2, characterized in that The file configuration information includes a file name and file access address information, and if it is determined that the metadata information of the target file data table in the multiple file data tables successfully matches the file configuration information, the metadata information of the target file data table is stored in the file master data table, including: If it is determined that the metadata information of the target file data table in the multiple file data tables successfully matches at least the file name and the file access address information in the file configuration information, the metadata information of the target file data table is stored in the file master data table.

4. The method according to claim 1, characterized in that: The file information also includes a storage server type. If it is determined that there is a file in the target file master data table with the same identification information as the target file and a different file access address, before marking the file as a duplicate file corresponding to the target file, the method further includes: Based on the file access address and the storage server type, identification information of the target file is determined.

5. A document inspection system, characterized in that: The system includes a task scheduling module and a data quality module; The task scheduling module is used to send an execution instruction for the file inspection task to the data quality module when the start execution time of the file inspection task is monitored; The data quality module is used to determine the target file master data table when receiving the execution instruction for the file inspection task; wherein the target file master data table stores file information of the target file, the file information is obtained from multiple file data tables, the file information includes a file access address, and the file information is used to determine the identification information of the target file; The data quality module is also used to determine whether there is a file in the target file master data table that has the same identification information as the target file and a different file access address from the target file, and when it is determined that there is a file in the target file master data table that has the same identification information as the target file and a different file access address, the file is determined as a duplicate file corresponding to the target file.

6. The system according to claim 5, characterized in that The system also includes a data integration module; The task scheduling module is further used to send an execution instruction for the data integration task to the data integration module when the start execution time of the data integration task is monitored; The data integration module is used to scan the multiple file data tables when receiving an execution instruction for the data integration task; wherein the data integration task carries file configuration information, and the file configuration information is used to determine the data table containing the file information; The data integration module is also used to match the metadata information of the scanned multiple file data tables with the file configuration information, and when it is determined that the metadata information of the target file data table in the multiple file data tables successfully matches the file configuration information, the metadata information of the target file data table is stored in the file master data table.

7. A document inspection device, characterized in that: The device comprises: A first determination module is used to determine a target file master data table in response to an execution instruction for a file checking task; wherein the target file master data table stores file information of the target file, the file information is obtained from a plurality of file data tables, the file information includes a file access address, and the file information is used to determine identification information of the target file; The marking module is used to mark the file as a duplicate file corresponding to the target file if it is determined that there is a file in the target file master data table that has the same identification information as the target file and a different file access address.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the method according to any one of claims 1 to 4.

9. A document inspection device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images