Data consistency checking method and device, computer device and medium

Abstract

The present disclosure relates to a data consistency checking method and device, computer equipment and medium; wherein the method comprises: obtaining a first to-be-checked task corresponding to a source database, wherein the first to-be-checked task comprises a plurality of to-be-checked sets; obtaining a second to-be-checked task corresponding to a target database; determining, from the plurality of to-be-checked sets included in the second to-be-checked task, a target to-be-checked set corresponding to each coroutine component according to the number of coroutine components and the target number of to-be-checked sets included in the second to-be-checked task; for each coroutine component in all coroutine components, performing data consistency checking on the first content and the second content of the target to-be-checked set corresponding to the current coroutine component by the current coroutine component to obtain a corresponding checking result. The present disclosure can perform data consistency checking on all to-be-checked sets in the to-be-checked task, which is more comprehensive and helps to improve the efficiency of checking.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of data processing, and particularly relates to a data consistency checking method and device, a computer device and a medium. BACKGROUND

[0002] With the rapid development of the Internet big data era, a series of data, which can be referred to as "data assets", will be generated in the operation and development process of an enterprise. The data assets are usually stored in a database. When the database is migrated or restored remotely, it is necessary to determine the consistency of the data in the source database and the target database (the database after migration or the database after backup).

[0003] The prior art usually checks the consistency of data by using design auxiliary materials such as table names, field names and entity-relationship diagrams. The workload is huge and the efficiency is low. At the same time, data processing personnel need to manually check and analyze the data set on the basis of understanding the data set. The quality of work is limited by the professional business ability of the data processing personnel, and the accuracy is low and the efficiency is not high.

[0004] With the increase of the data scale and the improvement of the complexity of the data assets, the prior art cannot meet the needs of data consistency checking. SUMMARY

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a data consistency checking method, device, computer device and medium.

[0006] In a first aspect, the present disclosure provides a data consistency checking method, comprising:

[0007] obtaining a first to-be-checked task corresponding to a source database, wherein the first to-be-checked task comprises a plurality of to-be-checked sets;

[0008] obtaining a second to-be-checked task corresponding to a target database, wherein the second to-be-checked task comprises a plurality of to-be-checked sets;

[0009] determining, according to the number of coroutine components and the target number of to-be-checked sets included in the second to-be-checked task, a target to-be-checked set corresponding to each coroutine component from the plurality of to-be-checked sets included in the second to-be-checked task;

[0010] For each coroutine component in all coroutine components, performing data consistency checking on a first content of a target to-be-checked set corresponding to the current coroutine component and a second content by the current coroutine component, to obtain a corresponding checking result, wherein the second content is a content contained in a to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task.

[0011] Optionally, the determining, according to the number of the coroutine components and the target number of the to-be-checked sets included in the second to-be-checked task, of the target to-be-checked set corresponding to each coroutine component from the to-be-checked sets included in the second to-be-checked task comprises:

[0012] determining the number of the coroutine components;

[0013] determining, according to a size relationship between the number of the coroutine components and the target number, of the target to-be-checked set corresponding to each coroutine component.

[0014] Optionally, the determining, according to the size relationship between the number of the coroutine components and the target number, of the target to-be-checked set corresponding to each coroutine component comprises:

[0015] if the number of the coroutine components is greater than the target number, determining, according to the target number, a preset number of target coroutine components from all the coroutine components, wherein the preset number is the same as the target number;

[0016] determining, according to an arrangement order of the to-be-checked sets included in the second to-be-checked task, a first target to-be-checked set corresponding to each target coroutine component respectively;

[0017] determining, according to a number of documents included in the first target to-be-checked set, a second target to-be-checked set assisted by a first idle coroutine component for checking from the first target to-be-checked set, wherein the first idle coroutine component is a coroutine component other than the target coroutine component among all the coroutine components.

[0018] Optionally, after the determining, according to the number of documents included in the first target to-be-checked set, of the second target to-be-checked set assisted by the first idle coroutine component for checking from the first target to-be-checked set, the method further comprises:

[0019] obtaining a total number of documents included in the second target to-be-checked set;

[0020] determining, according to the total number of documents and the number of the coroutine components, a number of documents assisted by the first idle coroutine component for checking;

[0021] determining a document checking end point of the second target to-be-checked set;

[0022] determining a document skipping checking point according to the number of documents and the document checking end point;

[0023] determining a starting document and an ending document of the document assisted by the first idle coroutine component for checking according to the document skipping checking point and the document checking end point.

[0024] Optionally, the method further comprises:

[0025] If the number of the coroutine components is less than or equal to the target number, a third target to-be-checked set corresponding to each coroutine component is determined respectively;

[0026] After all the documents included in the third target to-be-checked set are checked, the coroutine component that checks the third target to-be-checked set is determined as a second idle coroutine component;

[0027] According to the arrangement order of the remaining to-be-checked sets in the second to-be-checked task except the third target to-be-checked set, a fourth target to-be-checked set corresponding to the second idle coroutine component is determined.

[0028] Optionally, the data consistency check on the first content and the second content of the target to-be-checked set corresponding to the current coroutine component by the current coroutine component to obtain the corresponding check result comprises:

[0029] If the first content and the second content are inconsistent through the current coroutine component, the corresponding check result is determined as not passing.

[0030] Optionally, the method further comprises:

[0031] determining whether all the documents included in all the target to-be-checked sets are checked;

[0032] If yes, it is determined that all the target to-be-checked sets are checked, and the work of all the coroutine components is stopped.

[0033] In a second aspect, the present disclosure provides a data consistency checking device, comprising:

[0034] a first obtaining module configured to obtain a first to-be-checked task corresponding to a source database, wherein the first to-be-checked task includes a plurality of to-be-checked sets;

[0035] a second obtaining module configured to obtain a second to-be-checked task corresponding to a target database, wherein the second to-be-checked task includes a plurality of to-be-checked sets;

[0036] a set determining module configured to determine a target to-be-checked set corresponding to each coroutine component from the plurality of to-be-checked sets included in the second to-be-checked task according to the number of coroutine components and a target number of to-be-checked sets included in the second to-be-checked task;

[0037] The result obtaining module is configured to, for each of the coroutine components, perform data consistency check on the first content and the second content of the target to-be-checked set corresponding to the current coroutine component by the current coroutine component, to obtain a corresponding checking result, wherein the second content is content contained in a to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task.

[0038] Optionally, the set determining module comprises:

[0039] The number determining unit is configured to determine the number of the coroutine components.

[0040] The target set determining unit is configured to determine, according to the number of the coroutine components and the size relationship of the target number, the target to-be-checked set corresponding to each coroutine component.

[0041] Optionally, the target set determining unit is specifically configured to:

[0042] If the number of the coroutine components is greater than the target number, a preset number of target coroutine components are determined from all the coroutine components according to the target number, wherein the preset number is the same as the target number.

[0043] According to the arrangement order of the to-be-checked sets included in the second to-be-checked task, the first target to-be-checked set corresponding to each target coroutine component is determined.

[0044] According to the number of documents contained in the first target to-be-checked set, the second target to-be-checked set assisted by the first idle coroutine component for checking is determined from the first target to-be-checked set, wherein the first idle coroutine component is a coroutine component other than the target coroutine component among all the coroutine components.

[0045] Optionally, the apparatus further comprises a document determining module configured to, after determining the second target to-be-checked set assisted by the first idle coroutine component for checking from the first target to-be-checked set, obtain the total number of documents contained in the second target to-be-checked set; determine the number of documents assisted by the first idle coroutine component for checking according to the total number of documents and the number of the coroutine components; determine the document checking end point of the second target to-be-checked set; determine the document skipping checking point according to the number of documents and the document checking end point; and determine the starting document and the ending document of the document assisted by the first idle coroutine component for checking according to the document skipping checking point and the document checking end point.

[0046] Optionally, the target set determining unit is further specifically configured to:

[0047] If the number of the coroutine components is less than or equal to the target number, a third target to-be-checked set corresponding to each coroutine component is determined respectively;

[0048] After all the documents included in the third target to-be-checked set are checked, the coroutine component that checks the third target to-be-checked set is determined as a second idle coroutine component;

[0049] According to the arrangement order of the remaining to-be-checked sets in the second to-be-checked task except the third target to-be-checked set, a fourth target to-be-checked set corresponding to the second idle coroutine component is determined.

[0050] Optionally, the result obtaining module is specifically configured to:

[0051] For each coroutine component in all the coroutine components, if it is determined by the current coroutine component that the first content and the second content are inconsistent, the corresponding checking result is determined as not passing.

[0052] Optionally, the apparatus further includes:

[0053] The judging module is configured to determine whether all the documents included in all the target to-be-checked sets are checked.

[0054] The stopping module is configured to, if yes, determine that all the target to-be-checked sets are checked, and stop the work of all the coroutine components.

[0055] In a third aspect, the present disclosure further provides a computer device, including:

[0056] One or more processors;

[0057] A storage device configured to store one or more programs,

[0058] When the one or more programs are executed by the one or more processors, the one or more processors implement the data consistency checking method in any of the embodiments of the present disclosure.

[0059] In a fourth aspect, the present disclosure further provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the data consistency checking method in any of the embodiments of the present disclosure.

[0060] Compared with the prior art, the technical scheme provided by the embodiments of the present disclosure has the following advantages: first, a first to-be-checked task corresponding to a source database is acquired, then a second to-be-checked task corresponding to a target database is acquired, then according to the number of coroutine components and the target number of to-be-checked sets included in the second to-be-checked task, a target to-be-checked set corresponding to each coroutine component is determined from a plurality of to-be-checked sets included in the second to-be-checked task, and finally, for each coroutine component in all coroutine components, the first content and the second content of the target to-be-checked set corresponding to the current coroutine component are checked by the current coroutine component for data consistency, and the corresponding checking result is obtained, wherein the second content is the content included in the to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task. Through the above method, data consistency checking can be performed on all to-be-checked sets in the to-be-checked task, which is more comprehensive and helps to improve the efficiency of the checking. BRIEF DESCRIPTION OF DRAWINGS

[0061] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate preferred embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the accompanying drawings required to be used in the embodiments or the prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0063] Figure 1 is a flow diagram of a data consistency checking method provided by an embodiment of the present disclosure;

[0064] Figure 2 is a flow diagram of another data consistency checking method provided by an embodiment of the present disclosure;

[0065] Figure 3 is a structural diagram of a data consistency checking device provided by an embodiment of the present disclosure;

[0066] Figure 4 is a structural diagram of a computer device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0067] In order to more clearly illustrate the above-mentioned purposes, features and advantages of the present disclosure, the schemes of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0068] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present disclosure, but the present disclosure can also be implemented in other manners different from those described herein; obviously, the embodiments described in the specification are only a part of the embodiments of the present disclosure, and not all the embodiments.

[0069] Figure 1 is a flow diagram of a data consistency checking method provided by an embodiment of the present disclosure. The embodiment can be applicable to the case of checking the consistency of data in a source database and a target database when the database is migrated or the database is recovered at a different place. The embodiment method can be executed by a data consistency checking device, which can be implemented in a hardware and / or software manner and can be configured in a computer device; the computer device can be a server.

[0070] As shown in Figure 1 , the method specifically includes the following:

[0071] S110, obtaining a first to-be-checked task corresponding to a source database, wherein the first to-be-checked task includes a plurality of to-be-checked sets.

[0072] The source database can be a database before migration or a database before database recovery at a different place. The source database includes but is not limited to an instance database. The first to-be-checked task can include a to-be-checked database corresponding to the source database (for example, a database A), to-be-checked sets included in each to-be-checked database, and documents included in each to-be-checked set, etc. The to-be-checked database can include the name of the to-be-checked database in the source database, the number of to-be-checked databases, and other information about the to-be-checked database, etc. The to-be-checked set can include the name of the to-be-checked set in the to-be-checked database in the source database, the number of to-be-checked sets, and other information about the to-be-checked set, etc. The to-be-checked database and the to-be-checked set included in each to-be-checked database in the first to-be-checked task can be determined according to actual conditions, or can be pre-set, for example, all or part of the databases in the first to-be-checked task can be used as to-be-checked databases, and all or part of the sets included in each to-be-checked database can be used as to-be-checked sets.

[0073] In order to check the consistency of data in the source database and the target database, a data consistency checking tool is written in Go language in the embodiment, and the data consistency is checked through the tool. When checking, the first to-be-checked task corresponding to the source database needs to be obtained, which can be read from the source database through the tool.

[0074] S120, obtaining a second to-be-checked task corresponding to a target database, wherein the second to-be-checked task includes a plurality of to-be-checked sets.

[0075] The target database can be a database obtained after migration or a database obtained after database off-site recovery. In general, the type of the target database is the same as the type of the source database. The second to-be-checked task can include to-be-checked databases corresponding to the target database (for example, database B), to-be-checked collections included in each to-be-checked database, and documents included in each to-be-checked collection, and the like. The to-be-checked database can include the names of to-be-checked databases in the target database, the number of to-be-checked databases, and other information about to-be-checked databases, and the like. The to-be-checked collection can include the names of to-be-checked collections in to-be-checked databases in the target database, the number of to-be-checked collections, and other information about to-be-checked collections, and the like. The to-be-checked databases and the to-be-checked collections included in each to-be-checked database in the second to-be-checked task can be determined according to actual conditions, or can be pre-set, for example, all or part of the databases in the second to-be-checked task can be to-be-checked databases, and all or part of the collections included in each to-be-checked database can be to-be-checked collections.

[0076] To perform consistency checking on the data in the source database and the target database, when the data consistency checking tool is used for checking in this embodiment, the second to-be-checked task corresponding to the target database needs to be obtained, and the second to-be-checked task can be read from the target database by using the tool.

[0077] S130, according to the number of coroutine components and the target number of to-be-checked collections included in the second to-be-checked task, determining, from the plurality of to-be-checked collections included in the second to-be-checked task, a target to-be-checked collection corresponding to each coroutine component.

[0078] The coroutine component can be understood as a program component providing concurrent operation capability set by the data consistency checking tool, and is mainly used for checking whether the content included in the second to-be-checked task and the corresponding content in the first to-be-checked task are consistent. The target to-be-checked collection can be understood as a to-be-checked collection to be checked by the coroutine component. The target number is the total number of to-be-checked collections included in the second to-be-checked task.

[0079] After the first to-be-checked task and the second to-be-checked task are obtained, according to the number of coroutine components and the target number of to-be-checked collections included in the second to-be-checked task, a target to-be-checked collection corresponding to each coroutine component can be allocated from the plurality of to-be-checked collections included in the second to-be-checked task, that is, the to-be-checked collection to be checked by each coroutine component can be determined, which can be that a plurality of coroutine components check one target to-be-checked collection, or one coroutine component checks one target to-be-checked collection, or other conditions, which are not specifically limited in this embodiment.

[0080] S140, for each coroutine component in all coroutine components, performing data consistency check on the first content in the target to-be-checked set corresponding to the current coroutine component and the second content in the target to-be-checked set corresponding to the current coroutine component by the current coroutine component, to obtain a corresponding check result, wherein the second content is the content contained in the to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task.

[0081] The first content can be understood as the content related to all documents contained in the target to-be-checked set corresponding to the current coroutine component, such as document name, document number, and content in the document, etc. The second content can be understood as the content related to all documents contained in the to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task.

[0082] After determining the target to-be-checked set corresponding to each coroutine component, since the number of coroutine components can be large, for each coroutine component in all coroutine components, the to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task is obtained according to the name, number or other information capable of representing the target to-be-checked set of the target to-be-checked set corresponding to the current coroutine component in all to-be-checked sets contained in the first to-be-checked task. Then, the first content contained in the target to-be-checked set corresponding to the current coroutine component and the second content contained in the to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task are checked for data consistency by the current coroutine component. Specifically, the first content and the second content can be checked for whether they are the same. After the data consistency check, the corresponding check result is obtained. Each coroutine component can correspond to a check result. Therefore, by summarizing all check results, the total check result, i.e., the check result of the first to-be-checked task and the second to-be-checked task, can be obtained.

[0083] In the embodiment, first, the first to-be-checked task corresponding to the source database is obtained, then the second to-be-checked task corresponding to the target database is obtained, and then the target to-be-checked set corresponding to each coroutine component is determined from the plurality of to-be-checked sets included in the second to-be-checked task according to the number of coroutine components and the target number of to-be-checked sets included in the second to-be-checked task. Finally, for each coroutine component in all coroutine components, the first content in the target to-be-checked set corresponding to the current coroutine component and the second content in the target to-be-checked set corresponding to the current coroutine component are checked for data consistency by the current coroutine component, to obtain a corresponding check result, wherein the second content is the content contained in the to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task. Through the above method, data consistency check can be performed on all to-be-checked sets in the to-be-checked task, which is more comprehensive and helps to improve the efficiency of the check.

[0084] In some embodiments, optionally, before the data consistency check on the first content and the second content of the target to-be-checked collection corresponding to the current coroutine component by the current coroutine component to obtain the corresponding check result, the method can further include:

[0085] For each coroutine component of all coroutine components, the name and the number of the to-be-checked database corresponding to the target database in the second to-be-checked task and the name and the number of the to-be-checked database corresponding to the source database in the first to-be-checked task are checked for data consistency by the current coroutine component, and the name and the number of the to-be-checked collection included in each to-be-checked database corresponding to the target database and the name and the number of the to-be-checked collection included in each to-be-checked database corresponding to the source database in the first to-be-checked task are checked for data consistency by the current coroutine component to obtain the initial check result.

[0086] In this embodiment, through the above-mentioned check, the accuracy of the data consistency check result can be further improved, and the process of data consistency check is more comprehensive, avoiding missing the contents of the check.

[0087] In some embodiments, optionally, the data consistency check on the first content and the second content of the target to-be-checked collection corresponding to the current coroutine component by the current coroutine component to obtain the corresponding check result can specifically include:

[0088] If it is determined by the current coroutine component that the first content and the second content are inconsistent, it is determined that the corresponding check result is failed.

[0089] Specifically, since the target to-be-checked collection and the to-be-checked collection corresponding to the target to-be-checked collection in the first to-be-checked task can contain a lot of content, that is, the content related to multiple documents, if the consistency check on the first content and the second content by the current coroutine component finds that the document content of at least one document is inconsistent, it is determined that the corresponding check result is failed.

[0090] In this embodiment, the method for determining that the check result is failed is simple and fast, which can save time.

[0091] Further, when it is determined that the corresponding check result is failed, the check result can be output, and the check result can include the number of documents in the target to-be-checked collection, the number of consistent documents, the number of inconsistent documents, and the identifier of the inconsistent document, such as the name of the inconsistent document or the identity document (Identity Document, ID) of the inconsistent document.

[0092] In the embodiment, when the check result is not passed, the number of documents in the target set to be checked, the number of consistent documents, the number of inconsistent documents, and the identifiers of the inconsistent documents are output, so that the user can quickly locate the inconsistent documents, the inconsistent documents can be modified, and the user experience is improved.

[0093] In some embodiments, optionally, the method can further specifically include:

[0094] determining whether all the documents contained in all the target sets to be checked are checked.

[0095] If yes, it is determined that all the target sets to be checked are checked, and the work of all the coroutine components is stopped.

[0096] In the embodiment, when the coroutine components perform data consistency check on the corresponding target sets to be checked and the corresponding sets to be checked in the first set to be checked, it is necessary to determine whether all the documents contained in all the target sets to be checked are checked. If all the documents are checked, it means that all the target sets to be checked are checked, and at this time, the work of all the coroutine components should be stopped to avoid wasting resources caused by the continuous running of the coroutine components. If there is a document that is not checked, it means that the target set to be checked is not checked, and at this time, the component for checking the document that is not checked needs to continue to check the document that is not checked until all the documents are checked, and then the work of all the coroutine components is stopped, so as to save resources.

[0097] Figure 2 is a flowchart of another data consistency check method provided by the embodiment of the disclosure. The embodiment is an optimization based on the above-mentioned embodiment. Optionally, the embodiment specifically explains the process of determining the target set to be checked corresponding to each coroutine component. As shown in Figure 2 the method specifically includes the following:

[0098] S210, a first set to be checked corresponding to a source database is acquired, wherein the first set to be checked includes a plurality of sets to be checked.

[0099] S220, a second set to be checked corresponding to a target database is acquired, wherein the second set to be checked includes a plurality of sets to be checked.

[0100] S230, the number of coroutine components is determined.

[0101] The number of coroutine components can be set when the data consistency check tool is written, and the number can be determined according to actual needs, for example, can be 5, and the embodiment does not make specific limitation.

[0102] Specifically, the number of coroutine components can be determined by querying the code file corresponding to the data consistency checking tool.

[0103] In S240, according to the size relationship between the number of coroutine components and the target number of the to-be-checked sets included in the second to-be-checked task, a target to-be-checked set corresponding to each coroutine component is determined.

[0104] After the number of coroutine components is determined, the number of coroutine components and the target number of the to-be-checked sets included in the second to-be-checked task are compared to obtain the size relationship therebetween, and according to the size relationship, a target to-be-checked set corresponding to each coroutine component is determined, that is, according to the size relationship, a target to-be-checked set corresponding to the checking of each coroutine component is determined, so that each coroutine component has its own checking task, and the coroutine component in the idle state is avoided.

[0105] In S250, for each coroutine component in all coroutine components, the first content and the second content of the target to-be-checked set corresponding to the current coroutine component are subjected to data consistency checking by the current coroutine component, to obtain a corresponding checking result, wherein the second content is the content included in the to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task.

[0106] In this embodiment, first, the first to-be-checked task corresponding to the source database and the second to-be-checked task corresponding to the target database are obtained, then the number of coroutine components is determined, and then according to the size relationship between the number of coroutine components and the target number of the to-be-checked sets included in the second to-be-checked task, a target to-be-checked set corresponding to each coroutine component is determined, and finally for each coroutine component in all coroutine components, the first content and the second content of the target to-be-checked set corresponding to the current coroutine component are subjected to data consistency checking by the current coroutine component, to obtain a corresponding checking result. Through the size relationship between the number of coroutine components and the target number, each coroutine component is allocated a target to-be-checked set to be checked, which is simple and efficient, and can make all coroutine components have corresponding checking tasks, avoid waste of resources, and at the same time can perform data consistency checking on all to-be-checked sets in the to-be-checked task, which is beneficial to improve the efficiency of checking and the accuracy of checking results.

[0107] In this embodiment, the determining of the target to-be-checked set corresponding to each coroutine component according to the size relationship between the number of coroutine components and the target number can specifically include:

[0108] If the number of coroutine components is greater than the target number, a preset number of target coroutine components are determined from all coroutine components according to the target number, wherein the preset number is the same as the target number.

[0109] determine, according to the number of documents included in the first target to-be-checked set, a second target to-be-checked set assisted by the first idle coroutine component for checking from the first target to-be-checked set, wherein the first idle coroutine component is a coroutine component other than the target coroutine component among all coroutine components.

[0110] determine, according to the number of documents included in the first target to-be-checked set, a second target to-be-checked set assisted by the first idle coroutine component for checking from the first target to-be-checked set, wherein the first idle coroutine component is a coroutine component other than the target coroutine component among all coroutine components.

[0111] Specifically, if the number of coroutine components is greater than the target number, a preset number of coroutine components with the same target number are selected from all coroutine components as target coroutine components, so that each to-be-checked set included in the second to-be-checked task has a corresponding coroutine component for checking. Specifically, according to the arrangement order of the to-be-checked sets included in the second to-be-checked task, the first target to-be-checked set corresponding to each target coroutine component can be determined. Since the more documents included in the first target to-be-checked set, the more time-consuming the checking process may be, according to the number of documents included in the first target to-be-checked set, the second target to-be-checked set assisted by the first idle coroutine component for checking is determined from the first target to-be-checked set. Specifically, the first target to-be-checked set including the most documents in the first target to-be-checked set can be determined as the second target to-be-checked set assisted by the first idle coroutine component for checking.

[0112] For example, assuming that the number of coroutine components is 5, the target number of to-be-checked sets included in the second to-be-checked task is 4, and the number of documents included in each to-be-checked set is different, the first four of the coroutine components can be used as target coroutine components, which can be any four of the coroutine components, and the embodiment is not limited. Then, according to the arrangement order of the four to-be-checked sets, a corresponding first target to-be-checked set is assigned to each target coroutine component, for example, the first target to-be-checked set corresponding to the coroutine component 1 can be to-be-checked set 1, the first target to-be-checked set corresponding to the coroutine component 2 can be to-be-checked set 2, …, and the first target to-be-checked set corresponding to the coroutine component 4 can be to-be-checked set 4. Since the coroutine component 5 is an idle coroutine component, according to the number of documents included in the four to-be-checked sets, the to-be-checked set including the most documents can be used as the first target to-be-checked set corresponding to the coroutine component 5.

[0113] In the embodiment, when the number of coroutine components is greater than the target number, the target to-be-checked set corresponding to each target coroutine component is determined by the above method, which is simple and efficient, thereby facilitating the improvement of the efficiency of the subsequent data consistency checking process.

[0114] In the embodiment, after determining the second target set to be checked assisted by the first idle coroutine component from the first target set to be checked, the method further comprises:

[0115] obtaining a total number of documents in the second target set to be checked;

[0116] determining a number of documents to be checked by the first idle coroutine component according to the total number of documents and the number of coroutine components;

[0117] determining a document checking end point of the second target set to be checked;

[0118] determining a document skipping checking point according to the number of documents and the document checking end point;

[0119] determining a starting document and an ending document of the documents to be checked by the first idle coroutine component according to the document skipping checking point and the document checking end point.

[0120] The document checking end point is the total number of documents in the second target set to be checked when the second target set to be checked is checked for the first time, and the document checking end point is updated when the second target set to be checked is assisted to be checked, and the updating can be performed according to the document skipping checking point. The document skipping checking point can be understood as the number of documents that do not need to be checked by the first idle coroutine component.

[0121] Specifically, after determining the second target set to be checked assisted by the first idle coroutine component from the first target set to be checked, the total number of documents in the second target set to be checked is obtained, and the number of documents to be checked by the first idle coroutine component is determined by dividing the total number of documents by the number of coroutine components. Then, according to whether the second target set to be checked is checked for the first time or assisted to be checked, the document checking end point of the second target set to be checked is determined, and the document checking end point is subtracted from the number of documents to be checked by the first idle coroutine component to obtain the document skipping checking point. Finally, according to the document skipping checking point and the document checking end point, the starting document and the ending document of the documents to be checked by the first idle coroutine component can be determined, and the documents to be checked by the first idle coroutine component are all the documents from the starting document to the ending document. The starting document is the next document of the document corresponding to the document skipping checking point, and the ending document is the document corresponding to the document checking end point.

[0122] For example, assuming that the number of coroutine components is 5, namely coroutine component A, coroutine component B, coroutine component C, coroutine component D and coroutine component E, the target number is 1, namely: the to-be-checked set included in the second to-be-checked task is set 1, and the set 1 has 1 million documents. At this time, assuming that coroutine component A is the target coroutine component corresponding to checking set 1, then coroutine component B, coroutine component C, coroutine component D and coroutine component E are all first idle coroutine components and all assist in checking set 1. First, coroutine component B starts to assist in checking, determines that the number of documents to be assisted in checking is 1 million / 5=200,000, the document checking end point is 1 million, and the document skipping checking point is 1 million-200,000=800,000, so coroutine component B corresponds to checking the 800,001th to 1 millionth document, and updates the document checking end point to 800,000. Then, coroutine component C starts to assist in checking, determines that the number of documents to be assisted in checking is 1 million / 5=200,000, the document checking end point is 800,000, and the document skipping checking point is 800,000-200,000=600,000, so coroutine component C corresponds to checking the 600,001th to 800,000th document, and updates the document checking end point to 600,000. Next, coroutine component D starts to assist in checking, determines that the number of documents to be assisted in checking is 1 million / 5=200,000, the document checking end point is 600,000, and the document skipping checking point is 600,000-200,000=400,000, so coroutine component D corresponds to checking the 400,001th to 600,000th document, and updates the document checking end point to 400,000. Finally, coroutine component E starts to assist in checking, determines that the number of documents to be assisted in checking is 1 million / 5=200,000, the document checking end point is 400,000, and the document skipping checking point is 400,000-200,000=200,000, so coroutine component E corresponds to checking the 200,001th to 400,000th document, and updates the document checking end point to 200,000.

[0123] In this embodiment, by determining which documents in the second target to-be-checked set are assisted in checking by the first idle coroutine component, the work efficiency can be improved, and the first idle coroutine component and the coroutine component corresponding to checking the second target to-be-checked set do not conflict in the checking process, for example, checking the documents that have been checked or checking the same documents, thereby causing waste of resources.

[0124] In this embodiment, optionally, the determining of the target to-be-checked set corresponding to each coroutine component according to the size relationship between the number of coroutine components and the target number can further include:

[0125] If the number of coroutine components is less than or equal to the target number, the third target to-be-checked set corresponding to each coroutine component is determined.

[0126] After all the documents included in the third target to-be-checked set are checked, the coroutine component that checks the third target to-be-checked set is determined as a second idle coroutine component;

[0127] According to the arrangement order of the remaining to-be-checked sets in the second to-be-checked task except the third target to-be-checked set, a fourth target to-be-checked set corresponding to the second idle coroutine component is determined.

[0128] Specifically, if the number of coroutine components is less than or equal to the target number, a third target to-be-checked set corresponding to each coroutine component is respectively allocated, that is, one coroutine component corresponds to check one third target to-be-checked set. Since the number of documents included in each third target to-be-checked set can be different, the time of checking completion can also be different. When all the documents included in a certain third target to-be-checked set are checked, the coroutine component that checks the third target to-be-checked set is an idle coroutine component, and therefore the coroutine component that checks the third target to-be-checked set is determined as a second idle coroutine component. Then, when the number of coroutine components is less than the target number and the remaining to-be-checked sets in the second to-be-checked task except all the third target to-be-checked sets are multiple, according to the arrangement order of the remaining to-be-checked sets, preferably, the to-be-checked set with the highest arrangement order in the remaining to-be-checked sets is taken as a fourth target to-be-checked set corresponding to the second idle coroutine component, that is, the second idle coroutine component assists in checking the fourth target to-be-checked set. When the number of coroutine components is equal to the target number and the remaining third target to-be-checked sets in the second to-be-checked task except the third target to-be-checked set whose checking is completed are multiple, according to the arrangement order of the remaining third target to-be-checked sets, preferably, the third target to-be-checked set with the highest arrangement order in the remaining third target to-be-checked sets is taken as a fourth target to-be-checked set corresponding to the second idle coroutine component. Which documents in the fourth target to-be-checked set are checked by the second idle coroutine component can be determined according to the method described above, which is not repeated here.

[0129] In the embodiment, when the number of coroutine components is less than or equal to the target number, the target to-be-checked set corresponding to each target coroutine component is determined by the above method, which is simple and efficient, and can avoid the coroutine component being in an idle state, thereby facilitating the efficiency of the subsequent data consistency checking process.

[0130] Figure 3 is a structural schematic diagram of a data consistency checking device provided by the embodiment of the present disclosure; the device is configured in a computer device, and can implement the data consistency checking method described in any embodiment of the present application. The device specifically includes the following:

[0131] The first obtaining module 310 is configured to obtain a first to-be-checked task corresponding to a source database, wherein the first to-be-checked task comprises a plurality of to-be-checked sets;

[0132] The second obtaining module 320 is configured to obtain a second to-be-checked task corresponding to a target database, wherein the second to-be-checked task comprises a plurality of to-be-checked sets;

[0133] The set determining module 330 is configured to determine, according to a number of coroutine components and a target number of to-be-checked sets included in the second to-be-checked task, a target to-be-checked set corresponding to each coroutine component from the plurality of to-be-checked sets included in the second to-be-checked task.

[0134] The result obtaining module 340 is configured to, for each coroutine component in all coroutine components, perform data consistency checking on first content of a target to-be-checked set corresponding to a current coroutine component and second content contained in a to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task by the current coroutine component, to obtain a corresponding checking result.

[0135] In this embodiment, optionally, the set determining module 330 comprises:

[0136] The number determining unit is configured to determine the number of coroutine components.

[0137] The target set determining unit is configured to determine, according to a size relationship between the number of coroutine components and the target number, a target to-be-checked set corresponding to each coroutine component.

[0138] In this embodiment, optionally, the target set determining unit is specifically configured to:

[0139] If the number of coroutine components is greater than the target number, a preset number of target coroutine components are determined from all coroutine components according to the target number, wherein the preset number is the same as the target number.

[0140] According to an arrangement order of to-be-checked sets included in the second to-be-checked task, a first target to-be-checked set corresponding to each target coroutine component is determined.

[0141] According to a number of documents contained in the first target to-be-checked set, a second target to-be-checked set assisted by a first idle coroutine component for checking is determined from the first target to-be-checked set, wherein the first idle coroutine component is a coroutine component other than the target coroutine component in all coroutine components.

[0142] In the embodiment, optionally, the apparatus further comprises a document determination module configured to: after determining the second target set of documents to be checked by the first idle coroutine component, acquire a total number of documents included in the second target set of documents to be checked; determine a number of documents to be checked by the first idle coroutine component according to the total number of documents and the number of coroutine components; determine a document checking end point of the second target set of documents to be checked; determine a document skipping checking point according to the number of documents and the document checking end point; and determine a starting document and an ending document of the documents to be checked by the first idle coroutine component according to the document skipping checking point and the document checking end point.

[0143] In the embodiment, optionally, the target set determination unit is further configured to:

[0144] If the number of coroutine components is less than or equal to the target number, determine a third target set of documents to be checked corresponding to each coroutine component respectively.

[0145] After all the documents included in the third target set of documents to be checked are checked, determine the coroutine component that checks the third target set of documents to be checked as a second idle coroutine component.

[0146] According to an arrangement order of the remaining sets of documents to be checked in the second set of documents to be checked except the third target set of documents to be checked, determine a fourth target set of documents to be checked corresponding to the second idle coroutine component.

[0147] In the embodiment, optionally, the result acquisition module is configured to:

[0148] For each coroutine component in all coroutine components, if it is determined by the current coroutine component that the first content is inconsistent with the second content, determine that the corresponding checking result is failed. In the embodiment, optionally, the apparatus further comprises:

[0149] The judging module is configured to determine whether all the documents included in all the target sets of documents to be checked are checked.

[0150] The stopping module is configured to, if yes, determine that all the target sets of documents to be checked are checked, and stop the work of all the coroutine components.

[0151] The data consistency checking device provided by the embodiment of the present disclosure can firstly obtain the first to-be-checked task corresponding to the source database, then obtain the second to-be-checked task corresponding to the target database, and then determine the target to-be-checked set corresponding to each coroutine component from the plurality of to-be-checked sets included in the second to-be-checked task according to the number of coroutine components and the target number of to-be-checked sets included in the second to-be-checked task. Finally, for each coroutine component in all coroutine components, the first content and the second content of the target to-be-checked set corresponding to the current coroutine component are subjected to data consistency checking by the current coroutine component, and the corresponding checking result is obtained. The data consistency checking device can perform data consistency checking on all to-be-checked sets in the to-be-checked task, and is more comprehensive and beneficial to improving the checking efficiency.

[0152] The data consistency checking device provided by the embodiment of the present disclosure can execute the data consistency checking method provided by any embodiment of the present disclosure, and has the corresponding function modules and beneficial effects of the execution method.

[0153] Figure 4 is a structural schematic diagram of a computer device provided by the embodiment of the present disclosure. As shown in Figure 4 , the computer device includes a processor 410 and a storage device 420; the number of processors 410 in the computer device can be one or more, Figure 4 , the processor 410 in the computer device is taken as an example; the processor 410 and the storage device 420 in the computer device can be connected through a bus or other means, Figure 4 , the connection through the bus is taken as an example.

[0154] The storage device 420 is a kind of computer readable storage medium, which can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the data consistency checking method in the embodiment of the present disclosure. The processor 410 executes the software programs, instructions and modules stored in the storage device 420, thereby performing various function applications and data processing of the computer device, i.e. realizing the data consistency checking method provided by the embodiment of the present disclosure.

[0155] The storage 420 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; and the data storage area can store data created according to the use of the terminal and the like. In addition, the storage 420 can include a high-speed random access memory, and can also include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some examples, the storage 420 can further include a memory disposed remotely with respect to the processor 410, which can be connected to the computer device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0156] The computer device provided by the embodiment can be used to execute the data consistency checking method provided by any of the above embodiments, and has the corresponding functions and advantages.

[0157] The embodiment of the present disclosure also provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to implement the data consistency checking method provided by the embodiment of the present disclosure.

[0158] Of course, the computer executable instructions of the storage medium containing computer executable instructions provided by the embodiment of the present disclosure are not limited to the method operations as described above, and can also perform the related operations in the data consistency checking method provided by any of the embodiments of the present disclosure.

[0159] From the above description of the embodiments, those skilled in the art can clearly understand that the present disclosure can be realized by means of software and necessary universal hardware, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a FLASH memory, a hard disk or an optical disk, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present disclosure.

[0160] It is worth noting that in the above embodiment of the data consistency checking device, each unit and module included is only divided according to the function logic, but is not limited to the above division, as long as the corresponding function can be realized; in addition, the specific name of each functional unit is only for easy mutual distinction, and does not limit the protection scope of the present disclosure.

[0161] It has to be noted that, in the present document, relational terms are intended to encompass the various possible relationships between means or components or steps in a process. For example, the phrase "first means and second means" is intended to mean that a first means and a second means can be active together, alone, or in any combination, that one or more first means is / are an alternative to one or more second means, or in an "either / or" relationship, and / or that one or more first means and one or more second means are concurrent in some processes but not necessarily in others. In addition, the phrase "first means and second means" is not necessarily used in the same context, although it can be. The phrase "first means and second means" is also not necessarily used in the same sentence, although it can be. The phrase "first means and second means" is also not necessarily of the same category, although it can be. The phrase "first means and second means" is also not necessarily of the same meaning, although it can be.

[0162] The foregoing is merely illustrative of the various ways and specific embodiments in which the disclosure can be carried out. Numerous modifications can be made to these specific embodiments by those skilled in the art without departing from the spirit and scope of the disclosure. Accordingly, it is intended that all such variations be included within the scope of the disclosure, which is defined solely by the claims appended hereto.

Claims

1. A method of data consistency checking, characterized by, The method comprises: obtaining a first to-be-checked task corresponding to a source database, wherein the first to-be-checked task comprises a plurality of to-be-checked sets; obtaining a second to-be-checked task corresponding to a target database, wherein the second to-be-checked task comprises a plurality of to-be-checked sets; determining, according to a number of coroutine components and a target number of to-be-checked sets included in the second to-be-checked task, a target to-be-checked set corresponding to each coroutine component from the plurality of to-be-checked sets included in the second to-be-checked task; for each coroutine component in all coroutine components, performing data consistency checking on first content of a target to-be-checked set corresponding to the current coroutine component and second content included in the target to-be-checked set corresponding to the current coroutine component by the current coroutine component to obtain a corresponding checking result, wherein the second content is content included in a to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task; the first content is all document-related content included in the target to-be-checked set corresponding to the current coroutine component, and the second content is all document-related content included in the to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task; wherein the all document-related content includes but is not limited to document name, document number and content in the document; the determining, according to a number of coroutine components and a target number of to-be-checked sets included in the second to-be-checked task, a target to-be-checked set corresponding to each coroutine component from the plurality of to-be-checked sets included in the second to-be-checked task, comprises: determining the number of coroutine components; determining, according to a size relationship between the number of coroutine components and the target number, a target to-be-checked set corresponding to each coroutine component; the determining, according to a size relationship between the number of coroutine components and the target number, a target to-be-checked set corresponding to each coroutine component, comprises: if the number of coroutine components is greater than the target number, then determining a preset number of target coroutine components from all coroutine components according to the target number, wherein the preset number is the same as the target number; determining, according to an arrangement order of to-be-checked sets included in the second to-be-checked task, a first target to-be-checked set corresponding to each target coroutine component respectively; determining, according to a number of documents included in the first target to-be-checked set, a second target to-be-checked set assisted by a first idle coroutine component for checking from the first target to-be-checked set, wherein the first idle coroutine component is a coroutine component other than the target coroutine component among all coroutine components.

2. The method of claim 1, wherein, after the determining, from the first target to-be-checked set, a second target to-be-checked set assisted by a first idle coroutine component for checking, the method further comprises: obtaining a total number of documents included in the second target to-be-checked set; determining, according to the total number of documents and the number of coroutine components, a number of documents assisted by the first idle coroutine component for checking; determining a document checking end point of the second target to-be-checked set; determining a document skipping checking point according to the number of documents and the document checking end point; According to the document skip checkpoint and the document check end point, a start document and an end document of a document assisted by the first idle coroutine component for checking are determined.

3. The method of claim 1, wherein, Further comprising: If the number of coroutine components is less than or equal to the target number, a third target to-be-checked set corresponding to each coroutine component is determined respectively; After all the documents included in the third target to-be-checked set are checked, the coroutine component for checking the third target to-be-checked set is determined as a second idle coroutine component; According to the arrangement order of the remaining to-be-checked sets in the second to-be-checked task except the third target to-be-checked set, a fourth target to-be-checked set corresponding to the second idle coroutine component is determined.

4. The method of claim 1, wherein, The data consistency checking of the first content and the second content of the target to-be-checked set corresponding to the current coroutine component by the current coroutine component includes: If the first content and the second content are determined to be inconsistent by the current coroutine component, the corresponding check result is determined to be failed.

5. The method according to any one of claims 1 to 4, characterized in that, Further comprising: Whether all the documents included in all the target to-be-checked sets are checked is determined; If yes, it is determined that all the target to-be-checked sets are checked, and the work of all the coroutine components is stopped.

6. A data consistency checking apparatus characterized by comprising: The device includes: A first obtaining module is configured to obtain a first to-be-checked task corresponding to a source database, wherein the first to-be-checked task includes a plurality of to-be-checked sets; A second obtaining module is configured to obtain a second to-be-checked task corresponding to a target database, wherein the second to-be-checked task includes a plurality of to-be-checked sets; A set determining module is configured to determine, according to the number of coroutine components and a target number of to-be-checked sets included in the second to-be-checked task, a target to-be-checked set corresponding to each coroutine component from the plurality of to-be-checked sets included in the second to-be-checked task; A result obtaining module is configured to, for each of all the coroutine components, perform data consistency checking of a first content and a second content of a target to-be-checked set corresponding to the current coroutine component by a current coroutine component to obtain a corresponding check result, wherein the second content is a content included in a to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task; The first content is a content related to all the documents included in the target to-be-checked set corresponding to the current coroutine component, and the second content is a content related to all the documents included in the to-be-checked set corresponding to the target to-be-checked set in the first to-be-checked task; wherein the content related to all the documents includes but is not limited to document names, document numbers, and contents in the documents; The set determining module includes: A number determining unit is configured to determine the number of coroutine components; A target set determining unit is configured to determine, according to the number of coroutine components and the size relationship of the target number, a target to-be-checked set corresponding to each coroutine component. The target set determination unit is specifically configured to: if the number of the coroutine components is greater than the target number, determining a preset number of target coroutine components from all the coroutine components according to the target number, wherein the preset number is the same as the target number; determining a first target to-be-inspected set corresponding to each target coroutine component according to an arrangement order of a to-be-inspected set included in the second to-be-inspected task; and determining a second target to-be-inspected set assisted by a first idle coroutine component for inspection from the first target to-be-inspected set according to a number of documents included in the first target to-be-inspected set, wherein the first idle coroutine component is a coroutine component other than the target coroutine component among all the coroutine components.

7. A computer device, comprising: comprising: one or more processors; a memory device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the method as claimed in any one of claims 1-5.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method as claimed in any one of claims 1-5.

Images