Data verification methods, devices, equipment and storage media

Abstract

This application provides a data verification method, apparatus, device, and storage medium applicable to the big data field. The method includes: for each transaction data in a first message queue, obtaining verification rule information corresponding to the first message queue; determining the data to be processed indicated by the tag to be processed in the transaction data based on the tag to be processed; performing value conversion processing on the data to be processed in the transaction data based on the value conversion rule information of the tag to be processed, and storing the converted data as the target data of the transaction data in a second message queue corresponding one-to-one with the first message queue; for the target data in each second message queue, determining the target data to be matched among the target data in the remaining second message queues, and comparing the target data and the target data to be matched to obtain a comparison result. This embodiment improves the accuracy of data comparison by performing value conversion on the transaction data based on verification rule information.

Description

Technical Field

[0001] This application relates to big data technology, and in particular to a data verification method, apparatus, device and storage medium. Background Technology

[0002] Currently, with the continuous development of information technology, enterprise personnel often need to verify a large amount of business data in order to provide a data foundation for subsequent business analysis.

[0003] In related technologies, data verification is usually done manually, which not only consumes a lot of manpower, but also results in low verification efficiency and low accuracy. Summary of the Invention

[0004] This application provides a data verification method, apparatus, device, and storage medium to solve the problem of low verification accuracy caused by manual data verification in related technologies.

[0005] Firstly, this application provides a data verification method, including:

[0006] For each transaction data in the first message queue, the verification rule information corresponding to the first message queue is obtained, wherein the verification rule information includes a tag to be processed and value conversion rule information corresponding to the tag to be processed; the transaction data has a tag; based on the tag to be processed, the data to be processed indicated by the tag to be processed is determined in the transaction data;

[0007] Based on the value conversion rule information of the tag to be processed, the data to be processed in the transaction data is converted to obtain the converted data; and the converted data is stored as the target data of the transaction data in the second message queue that corresponds one-to-one with the first message queue; the value conversion rule information is used to indicate the processing method of the data to be processed;

[0008] For each target data in the second message queue, among the target data in the remaining second message queues, target data to be matched is determined, and the target data and the target data to be matched are compared to obtain the comparison result.

[0009] In some embodiments, the tag to be processed includes: a status tag; the value of the status tag is used to indicate whether the transaction data is in a completed state; the value conversion rule information corresponding to the status tag includes a value mapping table; the value mapping table is used to store conversion data that corresponds one-to-one with the data to be processed indicated by the status tag;

[0010] Based on the value conversion rule information of the tag to be processed, the data to be processed in the transaction data is subjected to value conversion processing to obtain the converted data, including:

[0011] Based on the data to be processed indicated by the status label, the corresponding transformed data is searched in the value mapping table, and the transformed data is used as the transformed data.

[0012] In some embodiments, the tag to be processed further includes: an amount tag; the value of the amount tag is used to indicate the transaction amount in the transaction data; the value conversion rule information corresponding to the amount tag includes a value conversion expression;

[0013] Based on the value conversion rule information of the tag to be processed, the data to be processed in the transaction data is subjected to value conversion processing to obtain the converted data, including:

[0014] The data to be processed corresponding to the amount tag is input into the value conversion expression corresponding to the amount tag, and the value conversion process is performed to obtain the converted data.

[0015] In some embodiments, the target data includes a transaction identifier; determining the target data to be matched from the target data in the remaining second message queues for each target data in the second message queue includes:

[0016] For each target data in the second message queue, the target data in the remaining second message queues whose transaction identifiers are the same as the transaction identifiers of the target data are determined as the target data to be matched.

[0017] In some embodiments, obtaining the verification rule information corresponding to the first message queue includes:

[0018] For each transaction data in the first message queue, obtain the filtering rule information corresponding to the first message queue. If it is determined that the transaction data meets the filtering rule information, obtain the verification rule information corresponding to the first message queue. The filtering rule information is used to indicate the conditions met by the transaction data that needs to be verified.

[0019] In some embodiments, before obtaining the verification rule information corresponding to the first message queue, the method further includes:

[0020] Obtain operation log records for each data source, and filter out transaction data from the operation log records based on preset keywords; wherein, the preset keywords include at least one or more of: add, delete, and update;

[0021] The transaction data from the data source is stored in a first message queue that corresponds one-to-one with the data source.

[0022] In some embodiments, comparing the target data and the target data to be matched to obtain a comparison result includes:

[0023] If it is determined that the target data is not the same as the target data to be matched, then the first memory value corresponding to the target data and the second memory value corresponding to the target data to be matched are obtained.

[0024] If the sum of the first memory value and the second memory value is less than or equal to a preset memory value, then the different data in the target data and the target data to be matched are marked respectively, and the marked target data and the marked target data to be matched are sent to the user to notify the user that the comparison results are inconsistent.

[0025] In some embodiments, the target data includes a transaction identifier; the method further includes:

[0026] If it is determined that the sum of the first memory value and the second memory value is greater than the preset memory value, the transaction identifier is sent to the user to notify the user that the comparison result is inconsistent.

[0027] Secondly, this application provides a data verification device, comprising:

[0028] The first acquisition unit is used to acquire the verification rule information corresponding to the first message queue for each transaction data in the first message queue, wherein the verification rule information includes a tag to be processed and value conversion rule information corresponding to the tag to be processed; the transaction data has a tag;

[0029] The first determining unit is configured to determine, based on the tag to be processed, the data to be processed indicated by the tag to be processed in the transaction data;

[0030] The processing unit is used to perform value conversion processing on the data to be processed in the transaction data based on the value conversion rule information of the tag to be processed, so as to obtain the converted data;

[0031] The first storage unit is used to store the converted data as the target data of the transaction data in a second message queue that corresponds one-to-one with the first message queue; the value conversion rule information is used to indicate the processing method of the data to be processed;

[0032] The second determining unit is used to determine the target data to be matched from the target data in the remaining second message queues for each target data in the second message queue.

[0033] The comparison unit is used to compare the target data and the target data to be matched to obtain the comparison result.

[0034] In some embodiments, the tag to be processed includes: a status tag; the value of the status tag is used to indicate whether the transaction data is in a completed state; the value conversion rule information corresponding to the status tag includes a value mapping table; the value mapping table is used to store conversion data that corresponds one-to-one with the data to be processed indicated by the status tag;

[0035] The processing unit is specifically used for:

[0036] Based on the data to be processed indicated by the status label, the corresponding transformed data is searched in the value mapping table, and the transformed data is used as the transformed data.

[0037] In some embodiments, the tag to be processed further includes: an amount tag; the value of the amount tag is used to indicate the transaction amount in the transaction data; the value conversion rule information corresponding to the amount tag includes a value conversion expression;

[0038] The processing unit is specifically used for:

[0039] The data to be processed corresponding to the amount tag is input into the value conversion expression corresponding to the amount tag, and the value conversion process is performed to obtain the converted data.

[0040] In some embodiments, the target data includes a transaction identifier; the second determining unit is specifically used for:

[0041] For each target data in the second message queue, the target data in the remaining second message queues whose transaction identifiers are the same as the transaction identifiers of the target data are determined as the target data to be matched.

[0042] In some embodiments, the first acquisition unit includes:

[0043] The first acquisition module is used to acquire the filtering rule information corresponding to the first message queue for each transaction data in the first message queue;

[0044] The second acquisition module is used to acquire the verification rule information corresponding to the first message queue if it is determined that the transaction data meets the filtering rule information; the filtering rule information is used to indicate the conditions that the transaction data to be verified meets.

[0045] In some embodiments, the apparatus further includes:

[0046] The second acquisition unit is used to acquire the operation log record of each data source before the first acquisition unit acquires the verification rule information corresponding to the first message queue;

[0047] A filtering unit is used to filter transaction data from the operation log records based on preset keywords; wherein the preset keywords include at least one or more of the following: add, delete, update;

[0048] The second storage unit is used to store the transaction data of the data source into a first message queue that corresponds one-to-one with the data source.

[0049] In some embodiments, the comparison unit includes:

[0050] The third acquisition module is used to acquire, if it is determined that the target data is not the same as the target data to be matched, the first memory value corresponding to the target data and the second memory value corresponding to the target data to be matched.

[0051] The processing module is configured to mark the different data in the target data and the target data to be matched respectively if it is determined that the sum of the first memory value and the second memory value is less than or equal to a preset memory value;

[0052] The first sending module is used to send the target data after marking and the target data to be matched after marking to the user, so as to notify the user that the comparison results are inconsistent.

[0053] In some embodiments, the target data includes a transaction identifier; the apparatus further includes:

[0054] The second sending module is used to send the transaction identifier to the user if it is determined that the sum of the first memory value and the second memory value is greater than the preset memory value, so as to notify the user that the comparison result is inconsistent.

[0055] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0056] Memory; memory for storing executable instructions of the processor;

[0057] The processor is configured to execute the method as described in any of the first aspects according to the executable instructions.

[0058] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0059] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the method described in any one of the first aspects.

[0060] This application provides a data verification method, apparatus, device, and storage medium. The method includes: for each transaction data in a first message queue, obtaining verification rule information corresponding to the first message queue, wherein the verification rule information includes a tag to be processed and value conversion rule information corresponding to the tag to be processed; the transaction data has a tag; based on the tag to be processed, determining the data to be processed indicated by the tag to be processed in the transaction data; based on the value conversion rule information of the tag to be processed, performing value conversion processing on the data to be processed in the transaction data to obtain converted data; and storing the converted data as the target data of the transaction data in a second message queue corresponding one-to-one with the first message queue; the value conversion rule information is used to indicate the processing method of the data to be processed; for the target data in each second message queue, determining the target data to be matched in the target data of the remaining second message queues, and comparing the target data and the target data to be matched to obtain a comparison result. In this embodiment, when data verification is required, the verification rule information corresponding to the first message queue where the transaction data is located is first obtained. Then, the transaction data is processed by value transformation based on the verification rule information, so that the two data to be compared can use the same value transformation rule, thereby improving the accuracy of subsequent data comparison. Furthermore, in this embodiment, the target data obtained from the value transformation process is also stored in a second message queue during data verification. That is, the original data is stored in the first message queue, and the value-transformed data is stored in the second message queue, so that subsequent data processing can also be performed based on the original data in the first message queue. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0062] Figure 1 A flowchart illustrating a data verification method provided in an embodiment of this application;

[0063] Figure 2 A schematic diagram illustrating a data verification method provided in an embodiment of this application;

[0064] Figure 3 A flowchart illustrating yet another data verification method provided in this application embodiment;

[0065] Figure 4 A flowchart illustrating another data verification method provided in this application embodiment;

[0066] Figure 5 This is a schematic diagram illustrating an application scenario for data replication provided in an embodiment of this application;

[0067] Figure 6 This is a schematic diagram of the structure of a data verification device provided in an embodiment of this application;

[0068] Figure 7 A schematic diagram of another data verification device provided in the embodiments of this application;

[0069] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0070] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0072] Currently, with the continuous development of information technology, more and more data can be stored in electronic devices for users to view, process, and perform other operations. For example, electronic devices can store transaction record data, and users can verify the accuracy of the transaction record data later, so that they can conduct product demand analysis and product recommendations based on the transaction record data.

[0073] In related technologies, data verification is usually done manually. However, with the continuous increase in data volume, manual data verification requires a large amount of manpower and time, resulting in low verification efficiency, poor data accuracy, and affecting the timeliness of subsequent data processing based on the verified data.

[0074] To avoid at least one of the above-mentioned technical problems, the inventive concept of this application is as follows: when verifying the transaction data in the first message queue, this application will perform value conversion processing on the transaction data stored in the first message queue based on the verification rule information corresponding to the first message queue, and store the converted data as the target data in the second message queue corresponding to the first message queue, and then compare the data in the second message queue.

[0075] Based on the above-mentioned inventive concept, this application provides a data verification method, apparatus, device and storage medium, which can be applied to the field of big data to provide timeliness and accuracy of data verification.

[0076] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0077] Figure 1 This is a flowchart illustrating a data verification method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0078] S101. For each transaction data in the first message queue, obtain the verification rule information corresponding to the first message queue. The verification rule information includes the tag to be processed and the value conversion rule information corresponding to the tag to be processed. The transaction data has a tag.

[0079] For example, in this embodiment, the first message queue can be used to store data from the data source corresponding to the first message queue, and the data source and the first message queue can have a one-to-one correspondence. For instance, when data from both parties to a transaction is needed, the data from party A can be stored in the first message queue A, and the data from party B can be stored in the first message queue B.

[0080] Furthermore, each first message queue is pre-configured with corresponding verification rules, which include a pending label and corresponding value conversion rules. It is understandable that during data transactions, the two parties may use different representations for data with the same meaning, such as different date formats. Therefore, the transaction data in the first message queue needs to be processed based on the verification rules. Specifically, the pending label in the verification rules indicates the pending data in the transaction data of the corresponding first message queue; the value conversion rules corresponding to the pending label indicate the processing method for the pending data indicated by the pending label.

[0081] When data verification is required, for example, when an electronic device receives a data verification instruction sent by a user, or when the current time reaches the preset data verification time, or when new transaction data is detected in the first message queue, the verification rule information corresponding to the first message queue is first obtained for each transaction data in the first message queue, so that the transaction data in the first message queue can be processed based on the verification rule information.

[0082] In one example, when determining the verification rule information corresponding to the first message queue, the verification rule information has a unique information name, and the first message queue has a unique first message queue name. The verification rule information corresponding to the first message queue can be determined through the correspondence between the information name and the first message queue name.

[0083] It should be noted that this embodiment does not impose specific restrictions on the type of the first message queue; for example, it can be a Kafka message queue.

[0084] S102. Based on the pending label, determine the pending data indicated by the pending label in the transaction data.

[0085] For example, in this embodiment, each piece of data contained in each transaction data also has a label. When the label in the transaction data matches the label to be processed in the verification rule information, the data corresponding to the label in the transaction data can be used as the data to be processed indicated by the label to be processed.

[0086] For example, the data format of transaction data is as follows: "data; XXXX-XX-XX; Num; YYY;", where "data" and "Num" are tags in the transaction data, and "XXXX-XX-XX" is the data corresponding to the tag "data", and "YYY" is the data corresponding to the tag "Num". When the tag to be processed is "date", then "XXXX-XX-XX" can be used as the data to be processed.

[0087] S103. Based on the value conversion rule information of the tag to be processed, the data to be processed in the transaction data is converted to obtain the converted data, and the converted data is stored as the target data of the transaction data in the second message queue corresponding to the first message queue; the value conversion rule information is used to indicate the processing method of the data to be processed.

[0088] For example, in this embodiment, after the data to be processed is determined based on the tag to be processed in the verification rule information corresponding to the first message queue, the data to be processed can be transformed based on the value transformation rule information in the verification rule information to obtain the target data. Then, the processed target data is stored in the second message queue corresponding to the first message queue. It should be noted that the correspondence between the first message queue and the second message queue is one-to-one. That is, the first message queue can be used to store the original data, while the second message queue corresponding to the first message queue can be used to store the transformed transaction data.

[0089] S104. For the target data in each second message queue, determine the target data to be matched among the target data in the remaining second message queues, and compare the target data and the target data to be matched to obtain the comparison result.

[0090] For example, in this embodiment, after performing value conversion processing on the transaction data in the first message queue, for the target data in each second message queue, the target data to be matched is determined in the remaining second message queues, and then the target data and the target data to be matched are compared to obtain the comparison result.

[0091] In one example, since there is a one-to-one correspondence between the first message queue and the data source, and also a one-to-one correspondence between the first message queue and the second message queue, it can be determined that there is also a one-to-one correspondence between the second message queue and the data source. That is, when the two parties to a transaction need to compare data, they can first determine the second message queue corresponding to each party. After determining the two second message queues corresponding to the two parties, they can determine the two target data to be compared based on the transaction time of the transaction data in the first message queue corresponding to each target data in the second message queue. That is, the target data corresponding to the transaction data with the same transaction time in the two second message queues can be used as the target data to be compared, and thus the comparison result can be obtained.

[0092] For example, Figure 2 This is a schematic diagram illustrating a data verification method provided in an embodiment of this application. As shown, the figure includes four trading parties: party 1, party 2, party 3, and party 4. The transaction data of these four parties are stored in corresponding first message queues. Specifically, the transaction data of party 1 is stored in first message queue 1, the transaction data of party 2 is stored in first message queue 2, the transaction data of party 3 is stored in first message queue 3, and the transaction data of party 4 is stored in first message queue 4. Furthermore, first message queue 1 corresponds to second message queue 1; first message queue 2 corresponds to second message queue 2; first message queue 3 corresponds to second message queue 3; and first message queue 4 corresponds to second message queue 4. The second message queue stores the target data after value conversion processing of the transaction data in its corresponding first message queue.

[0093] When target data is stored in the second message queue, or a preset verification time is reached, or a verification instruction is received, for each second message queue, the target data in that second message queue can be compared with the target data to be matched corresponding to that target data, thereby obtaining a comparison result. For example, if transaction party 1 and transaction party 2 are two data sources that need to be verified, the target data in second message queue 1 can be matched with that in second message queue 2 and verified. Similarly, if transaction party 3 and transaction party 4 are two data sources that need to be verified, the target data in second message queue 3 can be matched with that in second message queue 4 and verified. Furthermore, in one embodiment, when matching target data in the second message queues corresponding to the two parties in a transaction, that is, when determining the target data to be matched corresponding to the target data in the other second message queues for the target data in a certain second message queue, it can be determined based on the transaction time of the transaction data corresponding to the target data. If the transaction time is the same, it can be used as the target data to be matched corresponding to that target data. Then, the target data is compared with its corresponding target data to be matched, thereby obtaining a comparison result.

[0094] Understandably, in this embodiment, when data verification is required, the verification rule information corresponding to the first message queue where the transaction data is located is first obtained. Then, the transaction data undergoes value transformation processing based on the verification rule information, so that the two data sets to be compared can use the same value transformation rule, thereby improving the accuracy of subsequent data comparisons. Furthermore, in this embodiment, during data verification, the target data obtained from the value transformation processing is also stored in a second message queue. That is, the original data is stored in the first message queue, and the value-transformed data is stored in the second message queue, so that subsequent data processing can also be performed based on the original data in the first message queue.

[0095] In some embodiments, the tag to be processed includes: a status tag; the value of the status tag is used to indicate whether the transaction data is in a completed state; the value conversion rule information corresponding to the status tag includes a value mapping table; the value mapping table is used to store the converted data that corresponds one-to-one with the data to be processed indicated by the status tag. Figure 2 Based on the illustrated embodiment, step S103 can be implemented in the following way:

[0096] Based on the data to be processed indicated by the status label, the corresponding transformation data is searched in the value mapping table, and the transformation data is used as the transformed data; the transformed data is used as the target data of the transaction data and stored in the second message queue that corresponds one-to-one with the first message queue.

[0097] For example, in this embodiment, the pending label in the verification rule information can be a status label. In transaction data, the status label can be used to indicate whether the transaction data has been completed, for example, whether the transferor has transferred the transaction amount, or whether the transfer recipient has successfully received the transfer amount. In this case, the value conversion rule information in the verification rule information can be a value mapping table. The value mapping table stores the conversion data that corresponds one-to-one with the pending data indicated by the status label, that is, the pending data before value conversion and its corresponding converted data.

[0098] Furthermore, when performing value transformation processing on the data to be processed corresponding to the status label in the transaction data, the transformation data corresponding to the data to be processed can be directly searched in the value transformation mapping table based on the data to be processed, and then the obtained transformation data is stored in the second message queue.

[0099] Understandably, the total number of possible values ​​for a status label is relatively small, typically including values ​​for both incomplete and completed transaction states. Therefore, a value mapping table can be set up to determine which value the status label in the transaction data needs to be converted to. By looking up the table, the converted data can be quickly determined.

[0100] In some embodiments, the tag to be processed further includes: an amount tag; wherein the value of the amount tag is used to indicate the transaction amount in the transaction data; the value conversion rule information corresponding to the amount tag includes a value conversion expression; based on the above embodiments, when executing step S103, it can be implemented in the following way:

[0101] Input the data to be processed corresponding to the amount tag into the value conversion expression corresponding to the amount tag, perform value conversion processing, obtain the converted data, and store the converted data as the target data in the second message queue that corresponds one-to-one with the first message queue.

[0102] For example, in this embodiment, when comparing transaction data, it is usually necessary to verify the transaction amount. Therefore, based on the above embodiments, this embodiment can also include an amount label in the pending label to indicate the transaction amount contained in the transaction data. It is understood that different parties may use different currency units for recording transaction amounts. In this case, to quickly unify the currency units of the transaction amounts between the two parties, the value conversion rule information in the verification rule information can use a value conversion expression to determine the converted transaction amount value.

[0103] It is understandable that in this embodiment, when there are many possible values ​​for the data to be processed corresponding to the tag to be processed, that is, when the value range is large, the value conversion rule can be set as a value conversion expression so that the target data after value conversion can be quickly calculated based on the value conversion expression and the data to be processed, thereby improving the efficiency of data verification.

[0104] Figure 3 A flowchart illustrating another data verification method provided in this application embodiment is shown below. Figure 3 As shown, the method includes the following steps:

[0105] S301. For each transaction data in the first message queue, obtain the filtering rule information corresponding to the first message queue. The filtering rule information is used to indicate the conditions that the transaction data to be verified must meet.

[0106] For example, in this embodiment, when data verification is required, for each transaction data in the first message queue, the filtering rule information corresponding to the first message queue can be obtained first, and then the transaction data can be determined based on the filtering rule information to determine whether the transaction data needs to be verified.

[0107] For example, when it is necessary to verify transaction data in the first message queue whose transaction amount is greater than the first preset value, the filtering rule information of the first message queue can be determined as the transaction amount being greater than the first preset value. Then, based on the filtering rule information, it can be determined whether the transaction data currently obtained in the first message queue needs to be verified.

[0108] S302. If it is determined that the transaction data meets the filtering rule information, then obtain the verification rule information corresponding to the first message queue; wherein, the verification rule information includes the tag to be processed and the value conversion rule information corresponding to the tag to be processed; the transaction data has tags.

[0109] For example, in this embodiment, when it is determined that the transaction data in the first message queue meets the filtering rule information corresponding to the first message queue, it indicates that the transaction data needs to be verified. At this time, the verification rule information corresponding to the first message queue is obtained again so as to perform value conversion on the transaction data that needs to be verified.

[0110] It is understood that in this embodiment, before performing value conversion processing on the transaction data in the first message queue, the transaction data can first be filtered based on the filtering rule information corresponding to the first message queue, so as to filter out the transaction data that needs to be verified, thereby improving the data verification efficiency.

[0111] S303. Based on the pending label, determine the pending data indicated by the pending label in the transaction data.

[0112] S304. Based on the value conversion rule information of the tag to be processed, the data to be processed in the transaction data is converted to obtain the converted data, and the converted data is stored as the target data of the transaction data in the second message queue corresponding to the first message queue; the value conversion rule information is used to indicate the processing method of the data to be processed.

[0113] For example, the specific principles of steps S303 and S304 in this embodiment can be found in steps S102 and S103, and will not be repeated here.

[0114] S305. For each target data in the second message queue, the target data in the remaining second message queues whose transaction identifiers are the same as the transaction identifiers of the target data are identified as target data to be matched. The target data includes the transaction identifier.

[0115] For example, in this embodiment, when determining the target data to be matched corresponding to the target data in each second message queue, it can be based on the transaction identifier contained in the target data. It is understood that the transaction identifier is a unique identifier used to indicate the current transaction when the transaction data is generated. Therefore, based on the transaction identifier in the target data, transaction data with the same transaction identifier can be searched in the target data of the remaining second message queues, and the target data with the same transaction identifier is determined as the target data to be matched.

[0116] It is understood that in this embodiment, when it is necessary to determine the target data to be matched corresponding to the target data, the target data to be checked and the target data to be matched can be determined based on the transaction identifier, so as to ensure that the target data and the target data to be matched are the two sets of data to be checked, thereby improving the accuracy of data verification.

[0117] S306. If it is determined that the target data is different from the target data to be matched, then obtain the first memory value corresponding to the target data and the second memory value corresponding to the target data to be matched.

[0118] For example, in this embodiment, when comparing the target data to be matched and the target data, if the comparison determines that the two are different, then the first memory value occupied by the target data and the second memory value of the target data to be matched can be determined.

[0119] S307. If the sum of the first memory value and the second memory value is less than or equal to the preset memory value, then the different data in the target data and the target data to be matched are marked respectively, and the marked target data and the marked target data to be matched are sent to the user to notify the user that the comparison results are inconsistent.

[0120] For example, after determining the first memory value of the target data and the second memory value of the target data to be matched, the first memory value and the second memory value are summed. If the summed value is less than or equal to a preset memory value, the different parts of the target data and the target data to be matched can be marked, for example, by highlighting, bolding, or underlining. Then, the processed target data and the processed target data to be matched are sent to the user so that the user can promptly identify the differences between the two target data. In addition, this embodiment also considers the memory value occupied by the target data and the target data to be matched when notifying the user. If the sum of the two memory values ​​is large, in order to avoid the data occupying a large amount of memory during the data transmission process and thus consuming a lot of transmission resources, it can be considered that the target data and the target data to be matched are transmitted only when the sum of the two memory values ​​is less than a preset memory value, so as to save bandwidth transmission resources.

[0121] S308. If it is determined that the sum of the first memory value and the second memory value is greater than the preset memory value, the transaction identifier is sent to the user to notify the user that the comparison result is inconsistent, and the target data includes the transaction identifier.

[0122] For example, in this embodiment, when the sum of the first memory value of the target data and the second memory value of the target data to be matched is greater than a preset memory value, the transaction identifier contained in the target data can be sent to the user so that the user can subsequently determine the transaction data with inconsistent comparison results in the message queue based on the obtained transaction identifier.

[0123] It is understood that in this embodiment, if the sum of the memory values ​​occupied by the target data and the target data to be matched is greater than the preset memory value, in order to save data transmission bandwidth, the transaction identifier corresponding to the target data can be directly sent to the user so that the user can quickly identify the data with inconsistent comparison results, which is convenient for the user to manually review or generate a comparison result error report.

[0124] In this embodiment, before performing value conversion processing on the transaction data in the first message queue, the transaction data can first be filtered based on the filtering rules corresponding to the first message queue to identify the transaction data that needs to be verified, thereby improving data verification efficiency. Furthermore, when a discrepancy is determined in the data comparison results, different methods can be selected to notify the user based on the difference between the sum of the first memory value of the target data and the second memory value of the target data to be matched and a preset memory value. When the sum is greater than the preset memory value, the transaction identifier contained in the target data can be sent directly. When the sum is less than the preset memory value, the target data and the target data to be matched can be sent directly to the user, and the data that do not match can be marked for easy viewing by the user. This method can save resources occupied by data transmission.

[0125] Figure 4 A flowchart illustrating another data verification method provided in this application embodiment is shown below. Figure 3 As shown, the method includes the following steps:

[0126] S401. Obtain the operation log records of each data source, and filter out the transaction data of the data source from the operation log records based on preset keywords; wherein, the preset keywords include at least one or more of the following: add, delete, update.

[0127] For example, in this embodiment, during data verification, the data corresponding to each data source stored in the database can be copied to the first message queue. When determining the data to be stored in the first message queue, the transaction data to be stored can be determined based on the operation log records corresponding to the data source. It is understood that the operation log records corresponding to the data source can be used to indicate added, deleted, and updated transaction data in the database of that data source. Therefore, for each data source's operation log records, transaction data can be searched in the operation log records based on preset keywords, where the preset keywords can include at least one or more of the terms "added," "deleted," and "updated."

[0128] S402. Store the transaction data from the data source into the first message queue that corresponds one-to-one with the data source.

[0129] For example, after filtering out the transaction data corresponding to the data source, the transaction data can be stored in the first message queue that corresponds one-to-one with the data source.

[0130] In some embodiments, when filtering transaction data of a data source in the operation log, all data corresponding to the data source can be extracted directly, or only the incremental data of the data source after a preset time can be extracted; no specific restrictions are imposed here.

[0131] It is understood that in this embodiment, before data verification, matching can be performed on the operation log records of each data source based on preset keywords, thereby quickly identifying the corresponding transaction data in the operation log records, thus improving the efficiency of data verification. In addition, the above processing method can also avoid the problem of low timeliness caused by users directly manually entering data.

[0132] In practical applications, Figure 5 This is a schematic diagram illustrating an application scenario for data replication provided in an embodiment of this application. The diagram includes a first data synchronization tool, a second data synchronization tool, a KAFKA message queue, a ZooKeeper service, and a control unit.

[0133] The first data synchronization tool (e.g., CDC for Oracle) can be used to execute step S401 above, that is, to determine the transaction data of the data source based on the operation log records of the database corresponding to the data source and preset keywords. The second data synchronization tool (e.g., CDC for Kafka) is used, under the control of the control unit, to acquire the transaction data determined by the first data synchronization tool, and then stores the acquired data in the Kafka message queue corresponding to the data source of the transaction data. In addition, the ZooKeeper service is also used to manage the Kafka cluster configuration when the second data synchronization tool stores transaction data in the Kafka message queue. The specific implementation principle can be found in the relevant technical principles, and will not be elaborated here.

[0134] S403. For each transaction data in the first message queue, obtain the verification rule information corresponding to the first message queue, wherein the verification rule information includes the tag to be processed and the value conversion rule information corresponding to the tag to be processed; the transaction data has a tag.

[0135] S404. Based on the pending label, determine the pending data indicated by the pending label in the transaction data.

[0136] S405. Based on the value conversion rule information of the tag to be processed, the data to be processed in the transaction data is converted, and the converted data is stored as the target data of the transaction data in the second message queue corresponding to the first message queue; the value conversion rule information is used to indicate the processing method of the data to be processed.

[0137] S406. For the target data in each second message queue, determine the target data to be matched among the target data in the remaining second message queues, and compare the target data and the target data to be matched to obtain the comparison result.

[0138] For example, the specific principles of steps S403-S406 in this embodiment can be found in steps S101-S104, and will not be repeated here.

[0139] Figure 6 This is a schematic diagram of the structure of a data verification device provided in an embodiment of this application, as shown below. Figure 6 As shown, the device includes:

[0140] The first acquisition unit 61 is used to acquire the verification rule information corresponding to the first message queue for each transaction data in the first message queue. The verification rule information includes the tag to be processed and the value conversion rule information corresponding to the tag to be processed. The transaction data has a tag.

[0141] The first determining unit 62 is used to determine the data to be processed indicated by the tag to be processed in the transaction data based on the tag to be processed.

[0142] The processing unit 63 is used to perform value transformation processing on the data to be processed in the transaction data based on the value transformation rule information of the tag to be processed, so as to obtain the transformed data.

[0143] The first storage unit 64 is used to store the converted data as the target data of the transaction into the second message queue that corresponds one-to-one with the first message queue; the value conversion rule information is used to indicate the processing method of the data to be processed.

[0144] The second determining unit 65 is used to determine the target data to be matched from the target data in the remaining second message queues for each target data in the second message queue.

[0145] The comparison unit 66 is used to compare the target data and the target data to be matched to obtain the comparison result.

[0146] The apparatus provided in this embodiment is used to implement the technical solution provided by the above method. Its implementation principle and technical effect are similar, and will not be described again.

[0147] Figure 7 This is a schematic diagram of the structure of another data verification device provided in the embodiments of this application. Figure 6 Based on the device shown, in this embodiment:

[0148] The tags to be processed include: status tags; the values ​​of the status tags are used to indicate whether the transaction data is in a completed state; the value conversion rule information corresponding to the status tags includes a value mapping table; the value mapping table is used to store the converted data that corresponds one-to-one with the data to be processed indicated by the status tags;

[0149] Processing unit 63 is specifically used for:

[0150] Based on the data to be processed indicated by the status label, the corresponding transformed data is found in the value mapping table, and the transformed data is used as the transformed data.

[0151] In some embodiments, the label to be processed further includes: an amount label; the value of the amount label is used to indicate the transaction amount in the transaction data; the value conversion rule information corresponding to the amount label includes a value conversion expression.

[0152] Processing unit 63 is specifically used for:

[0153] Input the data to be processed corresponding to the amount label into the value conversion expression corresponding to the amount label, perform value conversion processing, and obtain the converted data.

[0154] In some embodiments, the target data includes a transaction identifier; the second determining unit 65 is specifically used for:

[0155] For each target data in the second message queue, the target data in the remaining second message queues whose transaction identifiers are the same as the transaction identifiers of the target data are identified as the target data to be matched.

[0156] In some embodiments, the first acquisition unit 61 includes:

[0157] The first acquisition module 611 is used to acquire the filtering rule information corresponding to the first message queue for each transaction data in the first message queue.

[0158] The second acquisition module 612 is used to acquire the verification rule information corresponding to the first message queue if it is determined that the transaction data meets the filtering rule information; the filtering rule information is used to indicate the conditions that the transaction data to be verified meets.

[0159] In some embodiments, the apparatus further includes:

[0160] The second acquisition unit 67 is used to acquire the operation log record of each data source before the first acquisition unit 61 acquires the verification rule information corresponding to the first message queue.

[0161] The filtering unit 68 is used to filter transaction data from the operation log records based on preset keywords; wherein the preset keywords include at least one or more of the following: add, delete, update.

[0162] The second storage unit 69 is used to store the transaction data of the data source into the first message queue that corresponds one-to-one with the data source.

[0163] In some embodiments, the comparison unit 66 includes:

[0164] The third acquisition module 661 is used to acquire the first memory value corresponding to the target data and the second memory value corresponding to the target data to be matched if it is determined that the target data is different from the target data to be matched.

[0165] The processing module 662 is used to mark different data in the target data and the target data to be matched respectively if it is determined that the sum of the first memory value and the second memory value is less than or equal to a preset memory value.

[0166] The first sending module 663 is used to send the target data after marking and the target data to be matched after marking to the user, so as to notify the user that the comparison results are inconsistent.

[0167] In some embodiments, the target data includes a transaction identifier; the apparatus further includes:

[0168] The second sending module 664 is used to send a transaction identifier to the user if it is determined that the sum of the first memory value and the second memory value is greater than a preset memory value, so as to notify the user that the comparison results are inconsistent.

[0169] The apparatus provided in this embodiment is used to implement the technical solution provided by the above method. Its implementation principle and technical effect are similar, and will not be described again.

[0170] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 8 As shown, the electronic device includes:

[0171] The electronic device includes a processor 291 and a memory 292; it may also include a communication interface 293 and a bus 294. The processor 291, memory 292, and communication interface 293 can communicate with each other via the bus 294. The communication interface 293 can be used for information transmission. The processor 291 can invoke logical instructions stored in the memory 292 to execute the methods of the above embodiments.

[0172] Furthermore, the logic instructions in the aforementioned memory 292 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0173] The memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, thereby implementing the methods in the above-described method embodiments.

[0174] The memory 292 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

[0175] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the methods.

[0176] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements any one of the methods.

[0177] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0178] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data verification method, characterized in that, include: For each transaction data in the first message queue, the verification rule information corresponding to the first message queue is obtained, wherein the verification rule information includes a tag to be processed and value conversion rule information corresponding to the tag to be processed; the transaction data has a tag; based on the tag to be processed, the data to be processed indicated by the tag to be processed is determined in the transaction data; Based on the value conversion rule information of the tag to be processed, the data to be processed in the transaction data is converted to obtain the converted data; and the converted data is stored as the target data of the transaction data in the second message queue that corresponds one-to-one with the first message queue; the value conversion rule information is used to indicate the processing method of the data to be processed; For each target data in the second message queue, among the target data in the remaining second message queues, determine the target data to be matched, and compare the target data and the target data to be matched to obtain the comparison result; The comparison of the target data and the target data to be matched yields a comparison result, including: If it is determined that the target data is not the same as the target data to be matched, then the first memory value corresponding to the target data and the second memory value corresponding to the target data to be matched are obtained. The target data includes a transaction identifier; if it is determined that the sum of the first memory value and the second memory value is greater than a preset memory value, the transaction identifier is sent to the user to notify the user that the comparison result is inconsistent.

2. The method according to claim 1, characterized in that, The tag to be processed includes: a status tag; the value of the status tag is used to indicate whether the transaction data is in a completed state; the value conversion rule information corresponding to the status tag includes a value mapping table; the value mapping table is used to store the conversion data that corresponds one-to-one with the data to be processed indicated by the status tag; Based on the value conversion rule information of the tag to be processed, the data to be processed in the transaction data is subjected to value conversion processing to obtain the converted data, including: Based on the data to be processed indicated by the status label, the corresponding transformed data is searched in the value mapping table, and the transformed data is used as the transformed data.

3. The method according to claim 2, characterized in that, The tag to be processed further includes: an amount tag; the value of the amount tag is used to indicate the transaction amount in the transaction data; the value conversion rule information corresponding to the amount tag includes a value conversion expression; Based on the value conversion rule information of the tag to be processed, the data to be processed in the transaction data is subjected to value conversion processing to obtain the converted data, including: The data to be processed corresponding to the amount tag is input into the value conversion expression corresponding to the amount tag, and the value conversion process is performed to obtain the converted data.

4. The method according to claim 1, characterized in that, The target data includes a transaction identifier; the step of determining the target data to be matched from the target data in the remaining second message queues for each target data in the second message queue includes: For each target data in the second message queue, the target data in the remaining second message queues whose transaction identifiers are the same as the transaction identifiers of the target data are determined as the target data to be matched.

5. The method according to claim 1, characterized in that, Obtain the verification rule information corresponding to the first message queue, including: For each transaction data in the first message queue, obtain the filtering rule information corresponding to the first message queue. If it is determined that the transaction data meets the filtering rule information, obtain the verification rule information corresponding to the first message queue. The filtering rule information is used to indicate the conditions met by the transaction data that needs to be verified.

6. The method according to claim 1, characterized in that, Before obtaining the verification rule information corresponding to the first message queue, the method further includes: Obtain operation log records for each data source, and filter out transaction data from the operation log records based on preset keywords; wherein, the preset keywords include at least one or more of: add, delete, and update; The transaction data from the data source is stored in a first message queue that corresponds one-to-one with the data source.

7. The method according to any one of claims 1-6, characterized in that, Also includes: If the sum of the first memory value and the second memory value is less than or equal to a preset memory value, then the different data in the target data and the target data to be matched are marked respectively, and the marked target data and the marked target data to be matched are sent to the user to notify the user that the comparison results are inconsistent.

8. A data verification device, said data verification device being used to implement the data verification method according to any one of claims 1-7, characterized in that, include: The first acquisition unit is used to acquire the verification rule information corresponding to the first message queue for each transaction data in the first message queue, wherein the verification rule information includes a tag to be processed and value conversion rule information corresponding to the tag to be processed; the transaction data has a tag; The first determining unit is configured to determine, based on the tag to be processed, the data to be processed indicated by the tag to be processed in the transaction data; The processing unit is used to perform value conversion processing on the data to be processed in the transaction data based on the value conversion rule information of the tag to be processed, so as to obtain the converted data; The first storage unit is used to store the converted data as the target data of the transaction data in a second message queue that corresponds one-to-one with the first message queue; the value conversion rule information is used to indicate the processing method of the data to be processed; The second determining unit is used to determine the target data to be matched from the target data in the remaining second message queues for each target data in the second message queue. The comparison unit is used to compare the target data and the target data to be matched to obtain the comparison result.

9. An electronic device, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-7.

Images