Data verification method, device, electronic equipment and medium

Abstract

The embodiments of the present application provide a data verification method, device, electronic device and medium, and relate to the field of cloud storage technology. The embodiments of the present application include: after the data of the source storage node is fully migrated to the target storage node, compare whether the data information of the source storage node is consistent with the data information of the target storage node, if consistent, determine the number of data to be verified, and assign verification tasks to each sub-thread according to the number of data to be verified and the number of sub-threads, so that each sub-thread performs consistency verification on the data in the source storage node and the data in the target storage node based on its own verification task to obtain a consistency verification result. This method can take into account both verification efficiency and verification accuracy.

Description

Technical Field

[0001] The present invention relates to the field of cloud storage technology, and in particular to a data verification method, device, electronic equipment and medium. Background Art

[0002] In big data business scenarios, after all the data on the source database is migrated to the target database, you need to verify whether the data on the source database and the target database are consistent.

[0003] The verification method in the related art is to compare whether the number of data items in the database source end and the database target end are consistent. If the number of data items is consistent, it is determined that the data in the database source end and the database target end are consistent. However, the consistency of the number of data items does not mean that the data content is consistent. Using this method will lead to inaccurate verification results.

[0004] Alternatively, by sampling, some data in the database source end and the database target end are compared to see if they are consistent. If they are consistent, then all data in the database source end and the database target end are considered consistent. However, the unsampled data may also be inconsistent, and this method also has the problem of inaccurate verification results.

[0005] Alternatively, the data in the database source end and the database target end can be compared one by one. When it is determined that all the data in the database source end and the database target end are consistent, the data in the database source end and the database target end are considered consistent. However, the amount of data to be verified is large, and the verification efficiency is low by using the one-by-one comparison method. It can be seen that the related technology cannot take into account both verification efficiency and verification accuracy. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a data verification method, device, electronic device and medium to achieve both verification efficiency and verification accuracy. The specific technical solution is as follows:

[0007] In a first aspect, an embodiment of the present application provides a data verification method, comprising:

[0008] After all data of the source storage node is migrated to the target storage node, comparing whether the data information of the source storage node is consistent with the data information of the target storage node;

[0009] If they are consistent, the number of data to be verified is determined;

[0010] According to the amount of data to be verified and the number of sub-threads, a verification task is assigned to each sub-thread, so that each sub-thread, based on its own verification task, performs consistency verification on the data in the source storage node and the data in the target storage node to obtain a consistency verification result; wherein the verification task of each sub-thread includes a verification identifier of the data that the sub-thread needs to verify.

[0011] In a possible implementation, allocating a verification task to each sub-thread according to the amount of the data to be verified and the number of sub-threads includes:

[0012] Determine, according to the amount of the data to be verified and the amount of the sub-threads, a target amount of data to be allocated to each sub-thread on average;

[0013] According to the storage order of each data in the source storage node, the target amount of data is allocated to each sub-thread respectively.

[0014] In a possible implementation, after assigning a verification task to each sub-thread according to the amount of the data to be verified and the number of sub-threads, the method further includes:

[0015] The verification tasks assigned to each sub-thread are stored in a hash map respectively, so that each sub-thread obtains its own verification task from the hash map; wherein, for each sub-thread, the identifier of the sub-thread is a key value, and the verification identifier of the data that the sub-thread needs to verify is the value of the key value.

[0016] In a possible implementation, comparing whether the data information of the source storage node is consistent with the data information of the target storage node includes:

[0017] Comparing whether the amount of data in the source storage node is the same as the amount of data in the target storage node;

[0018] If the amount of data in the source storage node is the same as the amount of data in the target storage node, determining that the data information of the source storage node is consistent with the data information of the target storage node;

[0019] If the amount of data in the source storage node is different from the amount of data in the target storage node, it is determined that the data information of the source storage node is inconsistent with the data information of the target storage node.

[0020] In a possible implementation, the data information includes: at least one of the number of databases, the name of the database, the number of collections included in each database, the name of each collection, the number of documents included in each collection, and the number of indexes; comparing whether the data information of the source storage node is consistent with the data information of the target storage node includes:

[0021] The data information of the source storage node is compared with the data information of the target storage node one by one. If any one of them is different, it is determined that the data information of the source storage node is inconsistent with the data information of the target storage node; if all of them are the same, it is determined that the data information of the source storage node is consistent with the data information of the target storage node.

[0022] In a possible implementation manner, after comparing whether the data information of the source storage node is consistent with the data information of the target storage node, the method further includes:

[0023] If the data information of the source storage node is inconsistent with the data information of the target storage node, the difference information between the data information of the source storage node and the data information of the target storage node is recorded.

[0024] In a second aspect, an embodiment of the present application provides a data verification device, including:

[0025] A comparison module, used to compare whether the data information of the source storage node is consistent with the data information of the target storage node after the data of the source storage node is fully migrated to the target storage node;

[0026] a determination module, configured to determine the amount of data to be verified if the comparison result of the comparison module is that the data information of the source storage node is consistent with the data information of the target storage node;

[0027] An allocation module is used to allocate verification tasks to each sub-thread according to the amount of data to be verified and the number of sub-threads, so that each sub-thread, based on its own verification task, performs consistency verification on the data in the source storage node and the data in the target storage node to obtain a consistency verification result; wherein the verification task of each sub-thread includes a verification identifier of the data that the sub-thread needs to verify.

[0028] In a possible implementation, the allocation module is specifically configured to:

[0029] Determine, according to the amount of the data to be verified and the number of the sub-threads, a target amount of data to be allocated to each sub-thread on average;

[0030] According to the storage order of each data in the source storage node, the target amount of data is allocated to each sub-thread respectively.

[0031] In a possible implementation manner, the device further includes:

[0032] A storage module is used to store the verification tasks assigned to each sub-thread in a hash map respectively, so that each sub-thread obtains its own verification task from the hash map; wherein, for each sub-thread, the identifier of the sub-thread is a key value, and the verification identifier of the data that the sub-thread needs to verify is the value of the key value.

[0033] In a possible implementation, the comparison module is specifically configured to:

[0034] Comparing whether the amount of data in the source storage node is the same as the amount of data in the target storage node;

[0035] If the amount of data in the source storage node is the same as the amount of data in the target storage node, determining that the data information of the source storage node is consistent with the data information of the target storage node;

[0036] If the amount of data in the source storage node is different from the amount of data in the target storage node, it is determined that the data information of the source storage node is inconsistent with the data information of the target storage node.

[0037] In a possible implementation, the data information includes: at least one of the number of databases, the name of the database, the number of collections included in each database, the name of each collection, the number of documents included in each collection, and the number of indexes; the comparison module is specifically used to:

[0038] The data information of the source storage node is compared with the data information of the target storage node one by one. If any one of them is different, it is determined that the data information of the source storage node is inconsistent with the data information of the target storage node; if all of them are the same, it is determined that the data information of the source storage node is consistent with the data information of the target storage node.

[0039] In a possible implementation manner, the device further includes:

[0040] The recording module is used to record the difference information between the data information of the source storage node and the data information of the target storage node if the data information of the source storage node is inconsistent with the data information of the target storage node.

[0041] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0042] Memory, used to store computer programs;

[0043] The processor is used to implement any of the above-mentioned data verification method steps when executing the program stored in the memory.

[0044] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the data verification method described in the first aspect is implemented.

[0045] In a fifth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the data verification method described in the first aspect above.

[0046] Beneficial effects of the embodiments of the present application:

[0047] The data verification method, device, electronic device and medium provided by the embodiment of the present application can compare whether the data information of the source storage node and the target storage node are consistent after data migration. If they are consistent, the subsequent verification process can be performed to improve the verification efficiency. When the data information of the source storage node and the target storage node is consistent, a verification task can be assigned to each sub-thread according to the number of data to be verified and the number of sub-threads, so that each sub-thread performs a consistency verification on the data in the source storage node and the data in the target storage node based on its own verification task to obtain a consistency verification result. Verifying all data to be verified avoids the situation where some data is not verified, and ensures the accuracy of verification. And multiple sub-threads perform consistency verification, which can improve the efficiency of data verification, so the data verification method of the embodiment of the present application takes into account both verification efficiency and verification accuracy.

[0048] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1A schematic diagram of the structure of a database system provided by an embodiment of the present invention;

[0051] Figure 2 A flow chart of a data verification method provided by an embodiment of the present invention;

[0052] Figure 3 An exemplary schematic diagram of the relationship between a main thread, a hash map, and multiple sub-threads provided in an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the structure of a data verification device provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] In order to solve the problem that the verification efficiency and verification accuracy cannot be balanced in the related art, the embodiment of the present application provides a data verification method, which can be applied to a database system, such as Figure 1 As shown, the database system includes a source storage node, a target storage node and a verification server.

[0057] The verification server may be an independent server, or the verification server and the source storage node or the target storage node may be deployed in the same server. Figure 1 In the example, the verification server is taken as an independent server.

[0058] The verification server is used to perform consistency verification on the data in the source storage node and the target storage node after the data stored in the source storage node is fully migrated to the target storage node.

[0059] In the embodiment of the present application, the source storage node and the target storage node may be servers deployed with a MongoDB database. Of course, the type of database is not limited thereto, for example, it may also be a MySQL database, etc., which is not limited in the embodiment of the present invention. Among them, MongoDB is a database based on distributed file storage, which is a product between a relational database and a non-relational database. MySQL is a relational database management system.

[0060] The present invention provides a method for verifying data. Figure 1 The verification server in the verification server can specifically execute this method through the main thread, such as Figure 2 As shown, the method includes:

[0061] S201. After all data of a source storage node is migrated to a target storage node, compare whether data information of the source storage node is consistent with data information of the target storage node.

[0062] If they are consistent, S202 is executed; if they are inconsistent, the verification process is terminated.

[0063] S202: If they are consistent, determine the amount of data to be verified.

[0064] The amount of data to be verified is the amount of data stored in the source storage node or the amount of data stored in the target storage node.

[0065] S203. According to the amount of data to be verified and the number of sub-threads, a verification task is assigned to each sub-thread, so that each sub-thread performs consistency verification on the data in the source storage node and the data in the target storage node based on its own verification task to obtain a consistency verification result.

[0066] The verification task of each sub-thread includes a verification identifier of data that needs to be verified by the sub-thread.

[0067] In one implementation, the data to be verified may be distributed to each sub-thread in an even distribution manner, so that each sub-thread performs consistency verification on different data to be verified in parallel, and each sub-thread obtains a consistency verification result through the consistency verification.

[0068] In another embodiment, each sub-thread may be assigned a piece of data to be verified, so that each sub-thread performs consistency verification on the assigned piece of data to be verified, and obtains consistency verification results for the assigned piece of data to be verified. When any sub-thread completes the consistency verification, the next piece of data to be verified is assigned to the sub-thread, until all the data to be verified are verified.

[0069] The verification server can obtain the consistency verification result of each sub-thread, summarize the consistency verification result of each sub-thread, and output the summarized consistency verification result. If there is inconsistent data between the source storage node and the target storage node, the consistency verification result includes detailed information of the difference information, such as: the identification of the inconsistent data between the source storage node and the target storage node, the name of the collection to which the data belongs, and the name of the database to which it belongs.

[0070] The data verification method provided by the embodiment of the present invention can compare whether the data information of the source storage node and the target storage node are consistent after data migration. If they are consistent, the subsequent verification process can be performed to improve the verification efficiency. When the data information of the source storage node and the target storage node is consistent, a verification task can be assigned to each sub-thread according to the number of data to be verified and the number of sub-threads, so that each sub-thread performs a consistency verification on the data in the source storage node and the data in the target storage node based on its own verification task to obtain a consistency verification result. Verifying all data to be verified avoids the situation where some data is not verified, and ensures the accuracy of verification. And multiple sub-threads perform consistency verification, which can improve the efficiency of data verification, so the data verification method of the embodiment of the present application takes into account both verification efficiency and verification accuracy.

[0071] In the embodiment of the present application, the number of sub-threads can be set according to actual conditions. In one embodiment, in order to improve the verification speed, the number of sub-threads is less than or equal to the number of central processing unit (CPU) cores. Taking the number of sub-threads equal to the number of CPU cores as an example, each CPU core corresponds to one sub-thread, and the verification server can run multiple sub-threads in parallel through multiple CPU cores.

[0072] Optionally, before the verification server assigns a verification task to each sub-thread, the verification server may initialize a specified number of sub-threads, where the specified number is the above-mentioned number of sub-threads.

[0073] Initialization in the field of computer programming refers to assigning initial values ​​to data objects or variables. After the above initialization, the child thread can be used to check the consistency of the data.

[0074] If the verification task is allocated to each sub-thread in an even distribution manner, the above S203 may include:

[0075] Determine the target number of data to be allocated to each sub-thread on average according to the number of data to be verified and the number of sub-threads;

[0076] According to the storage order of each data in the source storage node, the target amount of data is allocated to each sub-thread.

[0077] The target number is the number of data to be verified divided by the number of sub-threads. If the result is not an integer, an approximate value is taken.

[0078] Exemplarily, the number of data to be verified is 1534570, and there are 16 sub-threads in total, then 1534570 / 16=95910.625, and 95911 can be used as the target number.

[0079] After determining the target number, assign verification tasks to each subthread according to the storage order of each data. Taking MongoDB as an example, each document can be regarded as a piece of data.

[0080] The verification tasks assigned to child thread 1 include: db1.col1.1_12347, db1.col2.1_21009, db1.col3.1_10000, db2.col1.1_52378, db2.col2.1_177, and the verification tasks assigned to child thread 2 include: db2.col2.178_62266, db3.col2.1_33822. Among them, the verification tasks listed above are all verification identifiers of data. If the embodiment of the present application is applied to different types of databases, the form of the verification identifier of the data is also different. In the embodiment of the present application, MongoDB is used as an example for explanation.

[0081] Where db represents database, col represents collection, db1.col1.1_12347 represents the 1st to 12347th documents in collection 1 of database 1. db2.col2.1_177 represents the 1st to 177th documents in collection 2 of database 2. According to the storage order, the last document to be verified allocated to child thread 1 is the 177th document in collection 2 of database 2.

[0082] Then, when assigning verification tasks to child thread 2, the assignment will start from the 178th document in collection 2 of database 2, and db2.col2.178_62266 represents the 178th to 62266th documents in collection 2 of database 2. If the number of unassigned documents to be verified is less than 95911 when assigning verification tasks to the last child thread, all unassigned documents to be verified will be assigned to the last child thread.

[0083] In an embodiment of the present application, after assigning a verification task to each sub-thread, the verification server can store the verification task assigned to each sub-thread in a hash map, so that each sub-thread obtains its own verification task from the hashmap.

[0084] For each sub-thread, the identifier of the sub-thread is a key value (key), and the verification identifier of the data that the sub-thread needs to verify is the value of the key value (value).

[0085] The relationship between the main thread, hashmap, and multiple sub-threads is as follows Figure 3As shown, after the verification server assigns verification tasks to each sub-thread through the main thread, the verification task is stored in the hashmap. Then the verification server starts each sub-thread through the main thread, and each sub-thread can obtain its own verification task from the hashmap and execute the verification task.

[0086] As an example, Figure 3 Three sub-threads are shown in FIG. 3 , but the number of sub-threads in the embodiment of the present invention is not limited thereto.

[0087] In combination with the example in the above embodiment, if the number of documents to be verified is 1534570, there are 16 sub-threads in total, and the identifiers of the sub-threads are worker1, worker2, worker3...worker16, then the verification tasks stored in the hashmap are as follows:

[0088] worker1:{db1.col1.1_12347,db1.col2.1_21009,db1.col3.1_10000,db2.col1.1_52378,db2.col2.1_177}

[0089] worker2:{db2.col2.178_62266,db3.col2.1_33822}

[0090] …

[0091] worker16:{db9.col1.3000_52266,db10.col2.1_1000,db10.col3.1_39000}.

[0092] In an implementation manner of the embodiment of the present application, the data information in the above S201 is the amount of data, and the above S201 may include:

[0093] Compare whether the amount of data in the source storage node is the same as the amount of data in the target storage node;

[0094] If the amount of data in the source storage node is the same as the amount of data in the target storage node, it is determined that the data information of the source storage node is consistent with the data information of the target storage node;

[0095] If the amount of data in the source storage node is different from the amount of data in the target storage node, it is determined that the data information of the source storage node is inconsistent with the data information of the target storage node.

[0096] By adopting this method, before performing consistency check on all data to be checked, the amount of data can be compared first. If the amount of data in the source storage node is different from the amount of data in the target storage node, it can be determined that the data stored in the source storage node is inconsistent with the data stored in the target storage node. There is no need to further perform consistency check on the content of the data to be checked, which can improve the verification efficiency.

[0097] In an embodiment of the present application, after the verification server compares whether the data information of the source storage node and the data information of the target storage node are consistent, if it is determined that the data information of the source storage node and the data information of the target storage node are inconsistent, the difference information between the data information of the source storage node and the data information of the target storage node is recorded.

[0098] For example, if after comparison it is determined that the amount of data in the source storage node is inconsistent with the amount of data in the target storage node, the recorded difference information may be the inconsistency of the total amount of data, or the recorded difference information may include: the amount of data in the source storage node and the amount of data in the target storage node.

[0099] The verification server may output the difference information as a verification result, so that a subsequent verification server or staff member can locate the cause of the inconsistency between the data information in the source storage node and the target storage node based on the difference information.

[0100] In another embodiment, the data information in the above S201 includes: at least one of the number of databases, the name of the database, the number of collections included in each database, the name of each collection, the number of documents included in each collection, and the number of indexes.

[0101] Based on this, the above S201 may include:

[0102] The data information of the source storage node is compared with the data information of the target storage node one by one. If any one of them is different, it is determined that the data information of the source storage node and the data information of the target storage node are inconsistent; if all of them are the same, it is determined that the data information of the source storage node and the data information of the target storage node are consistent.

[0103] For example, both the source storage node and the target storage node can be storage nodes deployed with MongoDB instances. That is to say, after the documents in the source MongoDB instance of the source storage node are fully migrated to the target MongoDB instance in the target storage node, the data information in the source MongoDB instance can be compared one by one with the data information in the target MongoDB instance.

[0104] In an embodiment of the present application, the verification server may compare each item of data information in the source storage node with that in the target storage node to determine whether there is a difference.

[0105] Alternatively, the verification server may compare one by one in a certain order. For example, first compare whether the number of databases is the same. If not, the verification process ends. Optionally, the difference information may be recorded as: the number of databases is different, and the number of databases in the source storage node and the number of databases in the target storage node may be further recorded.

[0106] If they are the same, for each database, compare whether the database name of the database in the source storage node is the same as that of the database in the target storage node. If they are different, end the verification process. Optionally, the difference information can be recorded as: the database name is different, and the name of the database in the source storage node and the name of the database in the target storage node can be further recorded.

[0107] If they are the same, for each database, compare the number of sets in the database in the source storage node with the number of sets in the database in the target storage node. If they are different, end the verification process. Optionally, the difference information can be recorded as: there is a difference in the number of sets. The name of the database, the number of sets included in the database in the source storage node, and the number of sets included in the database in the target storage node can also be further recorded.

[0108] If they are the same, then for each database, compare the set name of each set included in the database in the source storage node with the set name of the same set included in the database in the target storage node to see if they are the same. If they are different, then end the verification process. Optionally, the difference information can be recorded as: the set names are different, and the name of the database, the set name of the same set in the source storage node, and the set name in the target storage node can be further recorded. In other words, it is to record which set of which database in the source storage node and the target storage node has a different set name, as well as the set name of the set in the source storage node and the set name of the set in the target storage node. For example, the first set name of database 1 in the source storage node is A, and the first set name of database 1 in the target storage node is B.

[0109] If they are the same, for each set, compare the number of documents included in the set in the source storage node with the number of documents included in the set in the target storage node. If they are different, end the verification process. Optionally, the difference information can be recorded as: the number of documents is different. You can also further record the name of the set, the name of the database to which the set belongs, and the number of documents included in the set in the source storage node and the number of documents included in the set in the target storage node. For example, the first set of database 3 in the source storage node includes 100 documents, and the first set of database 3 in the target storage node includes 101 documents.

[0110] If they are the same, for each set, compare whether the number of indexes included in the set in the source storage node is the same as the number of indexes included in the set in the target storage node. If they are different, end the verification process. Optionally, the difference information can be recorded as: the number of indexes is different, and the name of the set, the name of the database to which the set belongs, and the number of indexes included in the set in the source storage node and the number of indexes included in the set in the target storage node can be further recorded. For example, the first set of database 3 in the source storage node includes 20 indexes, and the first set of database 3 in the target storage node includes 19 indexes.

[0111] If they are the same, it is determined that the data information of the source storage node is the same as the data information of the target storage node.

[0112] By adopting this comparison method, when any data information is found to be different, the verification is terminated, which can improve the verification efficiency.

[0113] Based on the same inventive concept, the present application also provides a data verification device, such as Figure 4 As shown, the device comprises:

[0114] A comparison module 401 is used to compare whether the data information of the source storage node is consistent with the data information of the target storage node after the data of the source storage node is fully migrated to the target storage node;

[0115] The determination module 402 is used to determine the amount of data to be verified if the comparison result of the comparison module 401 is that the data information of the source storage node is consistent with the data information of the target storage node;

[0116] The allocation module 403 is used to allocate verification tasks to each sub-thread according to the amount of data to be verified and the number of sub-threads, so that each sub-thread, based on its own verification task, performs consistency verification on the data in the source storage node and the data in the target storage node to obtain a consistency verification result; wherein the verification task of each sub-thread includes a verification identifier of the data that the sub-thread needs to verify.

[0117] Optionally, the allocation module 403 is specifically configured to:

[0118] Determine the target number of data to be allocated to each sub-thread on average according to the number of data to be verified and the number of sub-threads;

[0119] According to the storage order of each data in the source storage node, the target amount of data is allocated to each sub-thread.

[0120] Optionally, the device further comprises:

[0121] A storage module is used to store the verification tasks assigned to each sub-thread in a hash map, so that each sub-thread obtains its own verification task from the hash map; wherein, for each sub-thread, the identifier of the sub-thread is a key value, and the verification identifier of the data that the sub-thread needs to verify is the value of the key value.

[0122] Optionally, the comparison module 401 is specifically configured to:

[0123] Compare whether the amount of data in the source storage node is the same as the amount of data in the target storage node;

[0124] If the amount of data in the source storage node is the same as the amount of data in the target storage node, it is determined that the data information of the source storage node is consistent with the data information of the target storage node;

[0125] If the amount of data in the source storage node is different from the amount of data in the target storage node, it is determined that the data information of the source storage node is inconsistent with the data information of the target storage node.

[0126] Optionally, the data information includes: at least one of the number of databases, the name of the database, the number of collections included in each database, the name of each collection, the number of documents included in each collection, and the number of indexes; the comparison module 401 is specifically used to:

[0127] The data information of the source storage node is compared with the data information of the target storage node one by one. If any one of them is different, it is determined that the data information of the source storage node and the data information of the target storage node are inconsistent; if all of them are the same, it is determined that the data information of the source storage node and the data information of the target storage node are consistent.

[0128] Optionally, the device further comprises:

[0129] The recording module is used to record the difference information between the data information of the source storage node and the data information of the target storage node if the data information of the source storage node is inconsistent with the data information of the target storage node.

[0130] The embodiment of the present invention further provides an electronic device, which may be the verification server mentioned above. Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0131] Memory 503, used for storing computer programs;

[0132] The processor 501 is used to implement the method steps in the above method embodiment when executing the program stored in the memory 503.

[0133] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0134] The communication interface is used for communication between the above electronic device and other devices.

[0135] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0136] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0137] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data verification methods are implemented.

[0138] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any of the data verification methods in the above embodiments.

[0139] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0140] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0141] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data verification method, characterized in that: include: After all data of the source storage node is migrated to the target storage node, comparing whether the data information of the source storage node is consistent with the data information of the target storage node; If they are consistent, the number of data to be verified is determined; According to the amount of data to be verified and the number of sub-threads, a verification task is assigned to each sub-thread, so that each sub-thread, based on its own verification task, performs consistency verification on the data in the source storage node and the data in the target storage node to obtain a consistency verification result; wherein the verification task of each sub-thread includes a verification identifier of the data that the sub-thread needs to verify.

2. The method according to claim 1, characterized in that The allocating a verification task to each sub-thread according to the amount of the data to be verified and the number of sub-threads includes: Determine, according to the amount of the data to be verified and the amount of the sub-threads, a target amount of data to be allocated to each sub-thread on average; According to the storage order of each data in the source storage node, the target amount of data is allocated to each sub-thread respectively.

3. The method according to claim 2, characterized in that After allocating the verification task to each sub-thread according to the amount of the data to be verified and the number of sub-threads, the method further includes: The verification tasks assigned to each sub-thread are stored in a hash map respectively, so that each sub-thread obtains its own verification task from the hash map; wherein, for each sub-thread, the identifier of the sub-thread is a key value, and the verification identifier of the data that the sub-thread needs to verify is the value of the key value.

4. The method according to claim 1, characterized in that: The comparing whether the data information of the source storage node is consistent with the data information of the target storage node includes: Comparing whether the amount of data in the source storage node is the same as the amount of data in the target storage node; If the amount of data in the source storage node is the same as the amount of data in the target storage node, determining that the data information of the source storage node is consistent with the data information of the target storage node; If the amount of data in the source storage node is different from the amount of data in the target storage node, it is determined that the data information of the source storage node is inconsistent with the data information of the target storage node.

5. The method according to claim 1, characterized in that The data information includes: at least one of the number of databases, the name of the database, the number of collections included in each database, the name of each collection, the number of documents included in each collection, and the number of indexes; the comparing whether the data information of the source storage node is consistent with the data information of the target storage node includes: The data information of the source storage node is compared with the data information of the target storage node one by one. If any one of them is different, it is determined that the data information of the source storage node is inconsistent with the data information of the target storage node; if all of them are the same, it is determined that the data information of the source storage node is consistent with the data information of the target storage node.

6. The method according to any one of claims 1, 4 or 5, characterized in that After comparing whether the data information of the source storage node is consistent with the data information of the target storage node, the method further includes: If the data information of the source storage node is inconsistent with the data information of the target storage node, the difference information between the data information of the source storage node and the data information of the target storage node is recorded.

7. A data verification device, characterized in that: include: A comparison module, used to compare whether the data information of the source storage node is consistent with the data information of the target storage node after the data of the source storage node is fully migrated to the target storage node; a determination module, configured to determine the amount of data to be verified if the comparison result of the comparison module is that the data information of the source storage node is consistent with the data information of the target storage node; An allocation module is used to allocate verification tasks to each sub-thread according to the amount of data to be verified and the number of sub-threads, so that each sub-thread, based on its own verification task, performs consistency verification on the data in the source storage node and the data in the target storage node to obtain a consistency verification result; wherein the verification task of each sub-thread includes a verification identifier of the data that the sub-thread needs to verify.

8. The device according to claim 7, characterized in that The allocation module is specifically used for: Determine, according to the amount of the data to be verified and the number of the sub-threads, a target amount of data to be allocated to each sub-thread on average; According to the storage order of each data in the source storage node, the target amount of data is allocated to each sub-thread respectively.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.

Images