Data collation method and apparatus

Abstract

The application relates to the technical field of big data, in particular to a data checking method and device. The data checking method comprises the following steps: obtaining the identifier of a to-be-processed data file, the identifier of the to-be-processed data file comprising a first cache path of the to-be-processed data file in a first distributed file storage system and a second cache path of the to-be-processed data file in a second distributed file storage system, the second distributed file storage system being pre-synchronized with data files in the first distributed file storage system; obtaining first data information in the first distributed file storage system according to the first cache path; obtaining second data information in the second distributed file storage system according to the second cache path; and checking whether the first data information and the second data information are consistent to obtain a target checking result. The data checking method can effectively check data abnormalities between the synchronized distributed file storage systems, and guarantee the safety of the data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a data checking method and device. BACKGROUND

[0002] With the continuous development of the technical field of big data, a distributed file storage system becomes a frequently used technology in the Internet. For example, a plurality of small files are combined into a large file and stored in the distributed file storage system, and position information of each small file in the large file is written into a distributed database as an index, so that the number of files can be reduced and storage efficiency can be improved.

[0003] Due to the explosive growth of data volume and the growth of data value, enterprises need to implement data disaster recovery in the same city or in different places, that is, need to store data in the original distributed file storage system to other distributed file storage system through copying, synchronization and other means, in order to realize safe storage of data, high availability and business continuity. In the related art, there are various methods for synchronizing the distributed file storage system, which can realize data replication between distributed file storage systems.

[0004] However, the method for synchronizing the distributed file storage system in the prior art only focuses on the synchronization operation itself between the distributed file storage systems, and lacks a subsequent regular data checking mechanism, so that problems such as data loss, tampering, accidental deletion, synchronization exception and the like existing in the synchronized data cannot be found in time.

[0005] Therefore, how to effectively check the data exception between the synchronized distributed file storage systems becomes a technical problem to be solved urgently. SUMMARY

[0006] The present application provides a data checking method and device, which can effectively check the data exception between the synchronized distributed file storage systems.

[0007] In a first aspect, the present application provides a data checking method applied to a server, the method comprising: obtaining an identifier of a to-be-processed data file, the identifier of the to-be-processed data file comprising a first cache path of the to-be-processed data file in a first distributed file storage system and a second cache path of the to-be-processed data file in a second distributed file storage system, the second distributed file storage system being pre-synchronized with data files in the first distributed file storage system; obtaining first data information in the first distributed file storage system according to the first cache path, the first data information comprising at least one first sub-file and a first index of the at least one first sub-file, the first index being used to indicate a position of the at least one first sub-file in the first data information; obtaining second data information in the second distributed file storage system according to the second cache path, the second data information comprising at least one second sub-file and a second index of the at least one second sub-file, the second index being used to indicate a position of the at least one second sub-file in the second data information; checking whether the first data information and the second data information are consistent to obtain a target checking result.

[0008] In the method, the identifier of the to-be-processed data file is obtained, and then the first data information and the second data information are obtained, and the first data information and the second data information are checked to obtain the final checking result. The checking method can check data abnormalities between the synchronized distributed file storage systems in a timely and effective manner, and then facilitate subsequent timely repair of the data to ensure data security and consistency.

[0009] In combination with the first aspect, in a possible implementation manner, the checking whether the first data information and the second data information are consistent to obtain the target checking result comprises: checking, for the at least one first sub-file and the at least one second sub-file, whether bytes of the first sub-file and bytes of the second sub-file are consistent to obtain a first checking result; checking whether a sub-file corresponding to the first index and the first sub-file are consistent to obtain a second checking result; checking whether a sub-file corresponding to the second index and the second sub-file are consistent to obtain a third checking result; checking whether the first index and the second index are consistent to obtain a fourth checking result; and determining the target checking result according to the first checking result, the second checking result, the third checking result, and the fourth checking result.

[0010] In the implementation manner, the corresponding checking results are obtained through file checking, index-file checking, and index checking, and the final target checking result is determined comprehensively. The checking result obtained through such a checking manner is more accurate and fine, and then facilitates subsequent repair and other operations on the data.

[0011] With reference to the first aspect, in a possible implementation manner, the determining the target collation result according to the first collation result, the second collation result, the third collation result and the fourth collation result comprises: if the first collation result, the second collation result, the third collation result and the fourth collation result are all consistent, determining that the target collation result is consistent; if any one of the first collation result, the second collation result, the third collation result and the fourth collation result is inconsistent, determining that the target collation result is inconsistent.

[0012] In this implementation manner, the collation result obtained by comprehensively combining various intermediate processes is more accurate.

[0013] With reference to the first aspect, in a possible implementation manner, the method further comprises: recording the target collation result and an operation time of the recording the target collation result.

[0014] In this implementation manner, the final target collation result and time information are recorded, which facilitates subsequent personnel to check and repair data, and thus the consistency and security of data are ensured.

[0015] With reference to the first aspect, in a possible implementation manner, before the obtaining the identifier of the data file to be processed, the method further comprises: updating a preset index list according to the first index of the at least one first sub-file and the second index of the at least one second sub-file, the index list storing a first original index of the first distributed file storage system and a second original index of the second distributed file storage system.

[0016] In this implementation manner, the corresponding index record information in the database is updated according to the latest first index of the first distributed file storage system and the second index of the second distributed file storage system, which can effectively ensure the accuracy of information in the database, and thus the effectiveness and accuracy of subsequent collation data information are ensured, and finally the consistency and security of data between the synchronized distributed file storage systems are ensured.

[0017] With reference to the first aspect, in a possible implementation manner, the updating the preset index list according to the first index of the at least one first sub-file and the second index of the at least one second sub-file comprises: comparing the first index with the first original index for the first index of the at least one first sub-file and the second index of the at least one second sub-file; in a case where the first index is inconsistent with the first original index, updating the first original index to the first index; comparing the second index with the second original index; and in a case where the second index is inconsistent with the second original index, updating the second original index to the second index.

[0018] In this implementation manner, by comparing whether the current index in the distributed file storage system is consistent with the original index, the original index in the database is updated in a case where the current index is inconsistent with the original index, so as to ensure the accuracy of the information in the database, and further ensure the effectiveness and accuracy of subsequent checking of data information.

[0019] In the second aspect, the application provides a data checking apparatus, which comprises: an acquisition module, configured to acquire an identifier of a data file to be processed, the identifier of the data file to be processed comprising a first cache path of the data file to be processed in a first distributed file storage system and a second cache path of the data file to be processed in a second distributed file storage system, the second distributed file storage system being pre-synchronized with data files in the first distributed file storage system; the acquisition module is further configured to acquire first data information in the first distributed file storage system according to the first cache path, the first data information comprising at least one first sub-file and a first index of the at least one first sub-file, the first index being used to indicate a position of the at least one first sub-file in the first data information; the acquisition module is further configured to acquire second data information in the second distributed file storage system according to the second cache path, the second data information comprising at least one second sub-file and a second index of the at least one second sub-file, the second index being used to indicate a position of the at least one second sub-file in the second data information; and a checking module, configured to check whether the first data information and the second data information are consistent, to obtain a target checking result.

[0020] With reference to the second aspect, in a possible implementation manner, the checking module is specifically configured to: check, for the at least one first sub-file and the at least one second sub-file, whether the bytes of the first sub-file and the bytes of the second sub-file are consistent, to obtain a first checking result; check whether the sub-file corresponding to the first index and the first sub-file are consistent, to obtain a second checking result; check whether the sub-file corresponding to the second index and the second sub-file are consistent, to obtain a third checking result; check whether the first index and the second index are consistent, to obtain a fourth checking result; and determine the target checking result according to the first checking result, the second checking result, the third checking result, and the fourth checking result.

[0021] With reference to the second aspect, in a possible implementation manner, the apparatus further includes a determination module, which is configured to: if the first checking result, the second checking result, the third checking result, and the fourth checking result are all consistent, determine that the target checking result is consistent; and if any one of the first checking result, the second checking result, the third checking result, and the fourth checking result is inconsistent, determine that the target checking result is inconsistent.

[0022] With reference to the second aspect, in a possible implementation manner, the apparatus further includes a recording module, which is configured to record the target checking result and an operation time of the recording of the target checking result.

[0023] With reference to the second aspect, in a possible implementation manner, the apparatus further includes an updating module, which is configured to update a preset index list according to the first index of the at least one first sub-file and the second index of the at least one second sub-file, the index list storing a first original index of the first distributed file storage system and a second original index of the second distributed file storage system.

[0024] With reference to the second aspect, in a possible implementation manner, the updating module is specifically configured to: compare the first index and the first original index for the first index of the at least one first sub-file and the second index of the at least one second sub-file; in a case where the first index and the first original index are inconsistent, update the first original index to the first index; compare the second index and the second original index; and in a case where the second index and the second original index are inconsistent, update the second original index to the second index.

[0025] In a third aspect, the present application provides a server, comprising: a memory and a processor; the memory is configured to store program instructions; the processor is configured to invoke the program instructions in the memory to execute the method in the first aspect or any possible implementation manner thereof.

[0026] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are configured to be executed by a processor to implement the method in the first aspect or any possible implementation manner thereof.

[0027] In a fifth aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program, and the computer program is configured to be executed by a processor to implement the method in the first aspect or any possible implementation manner thereof. BRIEF DESCRIPTION OF DRAWINGS

[0028] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0029] Figure 1 An application scenario schematic diagram provided for an embodiment of the present application;

[0030] Figure 2 A system architecture schematic diagram provided for an embodiment of the present application;

[0031] Figure 3 A data checking method flowchart provided for an embodiment of the present application;

[0032] Figure 4 A method flowchart for updating file information to a database provided for an embodiment of the present application;

[0033] Figure 5 An overall framework schematic diagram of the data checking method provided for an embodiment of the present application;

[0034] Figure 6 A data checking device schematic diagram provided for an embodiment of the present application;

[0035] Figure 7 A device schematic diagram provided for another embodiment of the present application.

[0036] The above drawings have shown the specific embodiments of the present application, and the following will have more detailed description. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0037] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following exemplary embodiments are not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application, as detailed in the appended claims.

[0038] Figure 1 An application scenario diagram is provided for an embodiment of the present application, and the scenario includes a first distributed file storage system 101 and a second distributed file storage system 102.

[0039] The distributed file storage system is a frequently used technology in the Internet, for example, a plurality of small files are combined into a large file and stored in a distributed file storage system (such as a hadoop distributed filesystem (HDFS)), and the position information of each small file in the large file is written into a distributed database as an index, which can reduce the number of files and improve storage efficiency.

[0040] Optionally, the distributed file storage system in the present application can also be referred to as a distributed file combined storage system.

[0041] With the explosive growth of data volume, the value of data grows exponentially, and the same city or off-site disaster recovery has become the mainstream way for enterprises to realize safe storage, high availability, and business continuity of data.

[0042] A plurality of data files are stored in the first distributed file storage system 101, and for the consideration of data value and safety, the original data files in the first distributed file storage system 101 need to be stored in the second distributed file storage system 102 through copying, synchronization, or the like.

[0043] It should be noted that when the data files of the first distributed file storage system 101 are copied or synchronized to the second distributed file storage system 102, the present application is not limited to the second distributed file storage system 102, and the data files of the first distributed file storage system 101 can also be copied or synchronized to a plurality of distributed file storage systems.

[0044] In the related art, various distributed file storage system synchronization methods have been proposed, which realize data replication between distributed file storage systems, such as software synchronization, hardware synchronization, and self-synchronization.

[0045] However, the synchronization method of the prior art distributed file storage system only focuses on the synchronization operation itself between the distributed file storage systems, and lacks a subsequent periodic data checking mechanism, so that the problems such as data loss, tampering, accidental deletion, synchronization exception and the like existing in the synchronized data cannot be found in time.

[0046] Therefore, how to effectively check the data exception between the synchronized distributed file storage systems becomes a technical problem to be solved.

[0047] Therefore, the present application provides a data checking method, aiming to solve the above technical problems of the prior art.

[0048] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0049] In combination with the application scenarios shown in Figure 1 , the system architecture provided by an embodiment of the present application is shown in Figure 2 , which includes a first distributed file storage system 101, a second distributed file storage system 102, a file extraction component 103, a database 104 and a data checking component 105. Figure 2 Specifically, all data files of the first distributed file storage system 101 are synchronized in the second distributed file storage system 102, the file extraction component 103 extracts the directory list information of all data files of the first distributed file storage system 101 and the second distributed file storage system 102, compares the directory list information of the data files in the first distributed file storage system 101 and the second distributed file storage system 102 stored in the database 104, updates the directory list information in the database 104, and sets the corresponding record as unchecked; the data checking component 105 randomly selects all unchecked file records under a certain directory in the database 104, checks the file-file, index-file and index-index respectively, obtains the final checking result and records in the database, facilitates subsequent data repair and the like operations, and effectively guarantees the data security consistency between the distributed file storage systems.

[0050] Optionally, if the information in the first distributed file storage system 101 and the second distributed file storage system 102 after the data checking and recording is changed, the file extraction component 103 will continue to perform the information extraction and database 104 updating operations.

[0051]

[0052] ​The first distributed file storage system 101 and the second distributed file storage system 102 are distributed file storage systems that merge a plurality of small files into one large file and store the location information of each small file in the large file as an index.

[0053] It should be noted that the "file" mentioned in the embodiments of the present application refers to the large file after merging. The file stored in the system has the concept of directory, and the metadata of the small file is written into the file data when the file is stored. The metadata of the small file and its index correspond one by one, and the corresponding index can be obtained by querying the metadata information. The metadata refers to other information besides the file entity, such as file name, author, creation time, last modification time, size, etc., and also includes other custom information set by the author.

[0054] The file extraction component 103 is used to connect the first distributed file storage system 101 and the second distributed file storage system 102, and update all file information to the database 104.

[0055] The database 104 is equivalent to the database concept in the computer technology field, and its implementation methods include but are not limited to a relational database management system (My structured query language, MySQL), an "ORACLE" database system, etc. The database 104 stores the path of each file in the distributed file storage system, the last modification time of the file in the two distributed file storage systems, the file size, the last check result, and the last check time. The possible values of the last check result are not checked, consistent, inconsistent, and in check.

[0056] The function of the data check component 105 is to check whether the file data and the index of the first distributed file storage system 101 and the second distributed file storage system 102 are consistent. According to different objects of the check, the data check component is divided into three sub-components: a file check component, an index-file check component, and an index check component.

[0057] Optionally, the components in the system architecture can be deployed on the same server or on different servers.

[0058] In combination with Figure 1 the application scenarios shown in the figures and Figure 2 the system architecture shown in the figures, Figure 3 a data check method flowchart provided by an embodiment of the present application is applied to a server. As Figure 3 shown, the method provided by the embodiment of the present application includes S301, S302, S203, S304, and S305. The following will be described in detail Figure 3 the steps in the method shown.

[0059] S301, obtain an identifier of a to-be-processed data file, the identifier of the to-be-processed data file including a first cache path of the to-be-processed data file in a first distributed file storage system and a second cache path of the to-be-processed data file in a second distributed file storage system, the second distributed file storage system being pre-synchronized with data files in the first distributed file storage system.

[0060] Optionally, all data files in the first distributed file storage system are pre-stored in the second distributed file storage system through copying, synchronization or the like.

[0061] Further, the server obtains the identifier of the to-be-processed data file, that is, obtains the first cache path and the second cache path of the to-be-processed data file, and then obtains corresponding to-be-processed data information.

[0062] S302, obtain first data information in the first distributed file storage system according to the first cache path, the first data information including at least one first sub-file and a first index of the at least one first sub-file, the first index being used to indicate a position of the at least one first sub-file in the first data information.

[0063] That is, the first data information in the to-be-processed data information includes a plurality of first sub-files and corresponding first indexes.

[0064] For the first distributed file storage system, the data information about the first distributed file storage system stored in the database before the server obtains the first data information is updated according to the latest index and file list in the first distributed file storage system.

[0065] As an optional implementation, the preset index list is updated according to the first index of the at least one first sub-file, the preset index list storing a first original index of the first distributed file storage system. For example, the first original index is updated to the first index by comparing the first index with the first original index in a case where the first index is inconsistent with the first original index.

[0066] It can be understood that the above Figure 3The file extraction component 103 mentioned in the document is connected to the first distributed file storage system 101, scans the first distributed file storage system 101 to obtain a first index list, reads all actually existing files for each first index list, and compares the file list of the first index list with the file list in the database 104; for the files that exist in the database but actually do not exist, the corresponding first index record is deleted from the database 104; for the files that actually exist but do not exist in the database 104, the corresponding first index record is inserted into the database 104, and the last check result is set as uncheck; for the files that actually exist and exist in the database 104, the actual file last modification time and file size are checked whether they are consistent with the record in the database, if not, the corresponding record in the database 104 is updated to the actual file value, and the last check result is set as uncheck; for the files that actually exist and exist in the database 104, and the actual file last modification time and file size are consistent with the record in the database, the last check time in the database 104 is checked whether it has exceeded the preset expiration time, if it has exceeded, the last check result of the corresponding record is set as uncheck to trigger recheck; if it has not exceeded, no operation is performed.

[0067] As an example, Figure 4 A flow chart of a method for updating file information to a database is provided for an embodiment of the present application. As Figure 4 shown.

[0068] S401, scanning the entire first distributed file storage system.

[0069] S402, scanning each first index list.

[0070] S403, comparing with the data information stored in the database.

[0071] S4041, in the database and not actually existing in the first distributed file storage system.

[0072] S4042, deleting the first index record.

[0073] S4043, not in the database and actually existing in the first distributed file storage system.

[0074] S4044, inserting the first index record.

[0075] S4045, in the database and actually existing in the first distributed file storage system.

[0076] S4046, judging whether the file has changed, if yes, performing S4047, otherwise performing S4048.

[0077] S4047, updating the first index record.

[0078] S4048, judging whether the last checking time is over the preset time, if yes, executing S4047, otherwise executing S4049.

[0079] S4049, no operation.

[0080] S303, acquiring second data information in the second distributed file storage system according to the second cache path, the second data information comprising at least one second sub-file and a second index of the at least one second sub-file, the second index being used to indicate the position of the at least one second sub-file in the second data information.

[0081] That is, the second data information in the data information to be processed comprises a plurality of second sub-files and corresponding second indexes.

[0082] For the second distributed file storage system, the data information about the second distributed file storage system stored in the database before the server acquires the second data information is updated according to the latest index and file list in the second distributed file storage system.

[0083] As an optional implementation, the preset index list is updated according to the second index of the at least one second sub-file, the preset index list storing a second original index of the second distributed file storage system. For example, comparing the second index with the second original index, in the case that the second index is inconsistent with the second original index, the second original index is updated to the second index.

[0084] The specific implementation process and examples are the same as the above-mentioned introduction about the first distributed file storage system in step S302, which will not be repeated here.

[0085] S304, checking whether the first data information and the second data information are consistent to obtain a target checking result.

[0086] In this step, all un-checked file records under a certain index directory in the database 104 are randomly selected, and the corresponding checking result is set to be in checking.

[0087] Further, the file checking component in the data checking component 105 in the above-mentioned Figure 3 is called to check whether the files of the two distributed file storage systems are consistent. Specifically, for the at least one first sub-file and the at least one second sub-file, the bytes of the first sub-file and the bytes of the second sub-file are compared to obtain a first checking result, the first checking result comprising checking consistent and checking inconsistent.

[0088] The above-mentioned Figure 3The index-file component in the data verification component 105 in the first distributed file storage system and the second distributed file storage system respectively checks whether the files and corresponding indexes are consistent. Specifically, the subfile corresponding to the first index is checked for consistency with the first subfile, obtaining a second verification result, which includes a consistency and a mismatch. The subfile corresponding to the second index is checked for consistency with the second subfile, obtaining a third verification result, which includes a consistency and a mismatch.

[0089] As an example, for the first distributed file storage system, the file content of the first distributed file storage system is parsed to obtain the metadata and location information of each first sub-file; the first index of each first sub-file is obtained based on the metadata query, and the location information of its first index is checked to see whether it is consistent with the location information obtained by parsing the file content, to obtain a second verification result.

[0090] Call the above Figure 3 The index checking component in the data checking component 105 checks whether the indexes of the two distributed file systems are consistent. Specifically, the first index and the second index are checked to see if they are consistent, and a fourth checking result is obtained. The fourth checking result includes a consistency and a non-consistency.

[0091] As an example, the contents of files in the first distributed file storage system and the second distributed file storage system are parsed according to the index-file verification component to obtain all metadata of at least one first sub-file and at least one second sub-file; the first index and second index data corresponding to the first sub-file and the second sub-file in the two distributed file storage systems are queried according to the metadata, and the comparison is made to determine whether they are consistent, thereby obtaining a fourth verification result.

[0092] Furthermore, a target verification result is determined based on the first verification result, the second verification result, the third verification result, and the fourth verification result.

[0093] Specifically, if the first verification result, the second verification result, the third verification result and the fourth verification result are all consistent, the target verification result is determined to be consistent; if any one of the first verification result, the second verification result, the third verification result and the fourth verification result is inconsistent, the target verification result is determined to be inconsistent.

[0094] Finally, the target verification result and information such as the operation time of the target verification result are recorded in the database.

[0095] As an example, based on the above embodiment, Figure 5 This is a schematic diagram of the overall framework of the data verification method provided in one embodiment of the present application. Figure 5As shown, the first distributed file storage system 101 includes a plurality of first indexes and a plurality of first subfiles, the second distributed file storage system 102 includes a plurality of second indexes and a plurality of second subfiles, and the data check between the first distributed file storage system 101 and the second distributed file storage system 102 includes three processes, i.e., a file check between the plurality of first subfiles and the plurality of second subfiles, an index check between the plurality of first indexes and the plurality of second indexes, an index-file check between the plurality of first indexes and the plurality of first subfiles, and an index-file check between the plurality of second indexes and the plurality of second subfiles, to finally obtain a target check result.

[0096] The data check method provided in the application can effectively check the inconsistency of data between the synchronized distributed file storage systems, and subsequent data repair can be performed through other means to ensure the consistency and security of the data. The data check component is randomly selected to perform the index list, which can be deployed in multiple concurrent modes to improve the check speed.

[0097] Optionally, the embodiments of the application take two distributed file storage systems as an example, but the same scheme can be extended to data check between three or even more distributed file storage systems, which is not limited in the application.

[0098] Figure 6 The data check device 600 provided in the embodiments of the application is shown, and the device 600 includes an acquisition module 601, a check module 602, a determination module 603, a recording module 604, and an updating module 605.

[0099] The acquisition module 601 is configured to acquire an identifier of a data file to be processed, the identifier of the data file to be processed including a first cache path of the data file to be processed in a first distributed file storage system and a second cache path of the data file to be processed in a second distributed file storage system, the second distributed file storage system being pre-synchronized with data files in the first distributed file storage system. The acquisition module 601 is further configured to acquire first data information in the first distributed file storage system according to the first cache path, the first data information including at least one first subfile and a first index of the at least one first subfile, the first index being used to indicate a position of the at least one first subfile in the first data information. The acquisition module 601 is further configured to acquire second data information in the second distributed file storage system according to the second cache path, the second data information including at least one second subfile and a second index of the at least one second subfile, the second index being used to indicate a position of the at least one second subfile in the second data information. The check module 602 is configured to check whether the first data information and the second data information are consistent to obtain a target check result.

[0100] As an example, the device 600 can be used to perform the data check method.Figure 3 The method shown, for example, the acquisition module 601 is used for executing S301, S302 and S303, and the checking module 602 is used for executing S304.

[0101] In a possible implementation, the checking module 602 is specifically configured to: check, for the at least one first sub-file and the at least one second sub-file, whether the bytes of the first sub-file and the bytes of the second sub-file are consistent, to obtain a first checking result; check whether the sub-file corresponding to the first index and the first sub-file are consistent, to obtain a second checking result; check whether the sub-file corresponding to the second index and the second sub-file are consistent, to obtain a third checking result; check whether the first index and the second index are consistent, to obtain a fourth checking result; and determine the target checking result according to the first checking result, the second checking result, the third checking result and the fourth checking result.

[0102] In a possible implementation, the determining module 603 is configured to: if the first checking result, the second checking result, the third checking result and the fourth checking result are all consistent, determine that the target checking result is consistent; and if any one of the first checking result, the second checking result, the third checking result and the fourth checking result is inconsistent, determine that the target checking result is inconsistent.

[0103] In a possible implementation, the recording module 604 is configured to record the target checking result and an operation time of recording the target checking result.

[0104] In a possible implementation, the updating module 605 is configured to update a preset index list according to the first index of the at least one first sub-file and the second index of the at least one second sub-file, and the index list stores the first original index of the first distributed file storage system and the second original index of the second distributed file storage system.

[0105] In a possible implementation, the updating module 605 is specifically configured to: compare the first index with the first original index for the first index of the at least one first sub-file and the second index of the at least one second sub-file; update the first original index to the first index in a case where the first index and the first original index are inconsistent; compare the second index with the second original index; and update the second original index to the second index in a case where the second index and the second original index are inconsistent.

[0106] Figure 7 The device schematic diagram provided by another embodiment of the present application is shown. Figure 7 The device shown can be used to execute the method of any one of the preceding embodiments.

[0107] As Figure 7As shown, the apparatus 700 of the embodiment includes a memory 701, a processor 702, a communication interface 703, and a bus 704. The memory 701, the processor 702, and the communication interface 703 are communicatively connected to each other through the bus 704.

[0108] The memory 701 can be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 701 can store a program, and when the program stored in the memory 701 is executed by the processor 702, the processor 702 is configured to execute each step of the method shown in the above embodiment.

[0109] The processor 702 can be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, configured to execute related programs to implement each method shown in the embodiments of the present application.

[0110] The processor 702 can also be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the method of the embodiments of the present application can be completed by the integrated logic circuit of hardware in the processor 702 or the instruction in the form of software.

[0111] The processor 702 described above can also be a general-purpose processor, a digital signal processing (DSP), an ASIC, a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0112] The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing for execution, or executed by a combination of hardware and software modules in the code processing. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory, or an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the memory 701, and the processor 702 reads the information in the memory 701, and combines the hardware to complete the functions required to be executed by the units included in the apparatus of the present application.

[0113] The communication interface 703 can use, but is not limited to, a transceiver such as a transceiver to realize the communication between the device 700 and other devices or communication networks.

[0114] The bus 704 can include a path for transmitting information between the various components (for example, the memory 701, the processor 702, the communication interface 703) of the device 700.

[0115] It should be understood that the device 700 shown in the embodiments of the present application can be an electronic device, or can also be a chip configured in an electronic device.

[0116] It should be noted that the data checking method and device in the present application can be used in the field of big data, and can also be used in any field other than the field of big data. The application of the data checking method and device in the present application is not limited.

[0117] It should be understood that the term "and / or" in the present application is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " in the present application generally represents an "or" relationship between the front and rear associated objects, but it can also represent an "and / or" relationship. The specific meaning can be understood according to the context before and after.

[0118] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c can mean a, b, c, a-b, a-c, b-c or a-b-c, where a, b and c can be single or multiple.

[0119] It should be understood that in various embodiments of the present application, the size of the sequence number of the above processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0120] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the system, the device and the unit described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0122] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, another division mode can be used. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or in other forms.

[0123] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0124] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0125] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.

[0126] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A collation method of data, characterized by, Applied to a server, the method comprises: obtaining an identifier of a to-be-processed data file, the identifier of the to-be-processed data file comprising a first cache path of the to-be-processed data file in a first distributed file storage system and a second cache path of the to-be-processed data file in a second distributed file storage system, the second distributed file storage system being pre-synchronized with data files in the first distributed file storage system; obtaining first data information in the first distributed file storage system according to the first cache path, the first data information comprising at least one first sub-file and a first index of the at least one first sub-file, the first index being used to indicate a position of the at least one first sub-file in the first data information; obtaining second data information in the second distributed file storage system according to the second cache path, the second data information comprising at least one second sub-file and a second index of the at least one second sub-file, the second index being used to indicate a position of the at least one second sub-file in the second data information; checking whether the first data information and the second data information are consistent to obtain a target checking result; the checking whether the first data information and the second data information are consistent to obtain the target checking result comprises: checking, for the at least one first sub-file and the at least one second sub-file, whether bytes of the first sub-file and bytes of the second sub-file are consistent to obtain a first checking result; checking whether a sub-file corresponding to the first index is consistent with the first sub-file to obtain a second checking result; checking whether a sub-file corresponding to the second index is consistent with the second sub-file to obtain a third checking result; checking whether the first index and the second index are consistent to obtain a fourth checking result; determining the target checking result according to the first checking result, the second checking result, the third checking result and the fourth checking result; the determining the target checking result according to the first checking result, the second checking result, the third checking result and the fourth checking result comprises: if the first checking result, the second checking result, the third checking result and the fourth checking result are all consistent, determining that the target checking result is consistent; if any one of the first checking result, the second checking result, the third checking result and the fourth checking result is inconsistent, determining that the target checking result is inconsistent.

2. The method of claim 1, wherein, the method further comprises: recording the target checking result and an operation time of the recording the target checking result.

3. The method of claim 1, wherein, before the obtaining the identifier of the to-be-processed data file, the method further comprises: updating a preset index list according to the first index of the at least one first sub-file and the second index of the at least one second sub-file, the index list storing a first original index of the first distributed file storage system and a second original index of the second distributed file storage system.

4. The method of claim 3, wherein, The updating the preset index list according to the first index of the at least one first sub-file and the second index of the at least one second sub-file comprises: Comparing the first index and the first original index for the first index of the at least one first sub-file and the second index of the at least one second sub-file; In the case that the first index is inconsistent with the first original index, updating the first original index to the first index; Comparing the second index and the second original index; In the case that the second index is inconsistent with the second original index, updating the second original index to the second index.

5. A collating apparatus of data, characterized by comprising: Comprise: The acquisition module is used for acquiring the identification of the to-be-processed data file, and the identification of the to-be-processed data file comprises a first cache path of the to-be-processed data file in a first distributed file storage system and a second cache path of the to-be-processed data file in a second distributed file storage system, and the second distributed file storage system is pre-synchronized with data files in the first distributed file storage system; The acquisition module is further used for acquiring first data information in the first distributed file storage system according to the first cache path, and the first data information comprises at least one first sub-file and a first index of the at least one first sub-file, and the first index is used for indicating the position of the at least one first sub-file in the first data information; The acquisition module is further used for acquiring second data information in the second distributed file storage system according to the second cache path, and the second data information comprises at least one second sub-file and a second index of the at least one second sub-file, and the second index is used for indicating the position of the at least one second sub-file in the second data information; The collation module is used for collating whether the first data information and the second data information are consistent to obtain a target collation result; The collation module is specifically used for collating whether the bytes of the first sub-file and the bytes of the second sub-file are consistent for the at least one first sub-file and the at least one second sub-file to obtain a first collation result; Collating whether the sub-file corresponding to the first index and the first sub-file are consistent to obtain a second collation result; Collating whether the sub-file corresponding to the second index and the second sub-file are consistent to obtain a third collation result; Collating whether the first index and the second index are consistent to obtain a fourth collation result; The target collation result is determined according to the first collation result, the second collation result, the third collation result and the fourth collation result; The determination module is used for determining that the target collation result is collation consistent if the first collation result, the second collation result, the third collation result and the fourth collation result are all collation consistent, and determining that the target collation result is collation inconsistent if any one of the first collation result, the second collation result, the third collation result and the fourth collation result is collation inconsistent.

6. A server, characterized by Comprise: Memory and processor; The memory is used for storing program instructions; The processor is configured to invoke program instructions in the memory to perform the method of any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method of any one of claims 1 to 4.

8. A computer program product, characterised in that, The computer program product comprises a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 4.

Images