A real-time data verification method for distributed system

Abstract

This invention discloses a real-time data verification method for distributed systems. The method subscribes to messages from a resource computing interface, parses verification and triggering feature values, and uses a caching module to manage data change messages within the same business process execution, using the verification feature value as the key. When the triggering condition is met, the verification module retrieves the corresponding message from the cache for verification; if verification fails, an alarm is triggered. Compared to traditional offline data verification methods, this invention provides high-quality real-time verification of distributed data, ensuring verification quality without overly relying on the quality and completeness of test cases.

Description

Technical Field

[0001] The present invention relates to the field of distributed system data security verification, and in particular to a distributed system data real-time verification method. Background Art

[0002] Distributed systems are composed of multiple computers, which are geographically dispersed and can be spread across a unit, a city, a country, or even the world. The functions of the entire system are implemented on various nodes, so distributed systems have the distribution of data processing. In the face of continuously upgraded business models and complex business environments, it is necessary to rely on efficient data verification methods and automated verification tools.

[0003] At present, the industry can adopt offline verification solutions for large-scale business data verification. However, offline data verification methods cannot verify some real-time data, and are highly dependent on the quality and integrity of use cases, which will result in high use case manufacturing costs, no way to discover problems in real time, and there are differences between online and offline environments, so there is no way to ensure the quality of verification. Summary of the invention

[0004] The purpose of the present invention is to provide a distributed system data real-time verification method in order to solve the above problems.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] A distributed system data real-time verification method is used to verify the data source changes caused by calling services in different links of a business process, and any data source changes when executing the same business process need to be written with the same verification feature value for identification, and at the same time, the execution status field of at least one key service in the business process is pre-specified as the trigger feature value;

[0007] The real-time verification method comprises:

[0008] S1. The data sources to be checked involved in different services in the business process are respectively used as resource computing interfaces, and the messages of each resource computing interface are subscribed to. The data change records in each data source to be checked are obtained in real time, and a verification message is generated and sent to the parsing module as the data source of the parsing module;

[0009] S2. After receiving the verification messages sent asynchronously by different resource computing interfaces, the parsing module calls the corresponding parsing rules according to the message type of each verification message, and parses the verification feature value and the trigger feature value from each verification message, and then sends the verification feature value and the verification message itself in each verification message as a key-value pair to the cache module for storage; at the same time, the parsing module adds the verification feature value and the trigger feature value parsed from each verification message to the first trigger message and sends it to the trigger module;

[0010] S3. After receiving the key-value pair data sent by the parsing module, the cache module stores them by category according to the key, and merges all the verification messages with the same key into a verification message set with the verification feature value as the key;

[0011] S4. After receiving the first trigger message sent by the parsing module, the trigger module calls the corresponding trigger rule according to the type of the trigger feature value, and adds the verification feature value in the first trigger message to the second trigger message according to the trigger rule and sends it to the verification module;

[0012] S5. The verification module reads a verification message set with the verification feature value as a key from the cache module according to the verification feature value in the second trigger message, performs structured processing on all the verification messages in the verification message set to form structured data for verification; then, according to the type of the verification feature value, the verification rule of the corresponding type in the rule module is called to verify the structured data, and if the verification is wrong, an alarm message is sent to the alarm strategy module;

[0013] S6. After receiving the alarm message, the alarm strategy module issues an alarm according to the preset alarm strategy.

[0014] Preferably, in S1, the resource computing interface is any type of structured data.

[0015] Preferably, in S2, the verification characteristic value is carried in each original verification message, and the trigger characteristic value is unique to the verification message corresponding to the pre-specified key service.

[0016] Preferably, in S4, the triggering rules of the triggering module support customization.

[0017] Preferably, in S4, the triggering rule of the trigger module is: after receiving the first trigger message sent by the parsing module, delay for a preset time, wait until the data source changes caused by all services in a business process are executed and the respective verification messages are stored in the cache module, and then add the verification feature value in the first trigger message to the second trigger message and send it to the verification module to trigger the verification.

[0018] Preferably, in S5, the verification module is pre-installed with structured processing methods corresponding to different types of verification messages. When the second trigger message is received, the corresponding structured processing method is called for each type of verification message in the verification message set, and the information to be verified is extracted and converted into structured data for storage.

[0019] Preferably, in S5, the rule module pre-sets corresponding checking rules for each type of trigger feature value, and the checking rules are implemented by writing a configuration file.

[0020] Preferably, in S6, the alarm strategy includes two methods: alarming the person in charge individually and alarming multiple people in a group.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] The present invention subscribes to messages from the resource computing interface and parses the verification feature value and the trigger feature value therefrom. The cache module uses the verification feature value as a key to merge and manage the data change messages during the same business process execution process, so that when the trigger condition is met, the verification module extracts the corresponding message from the cache for verification, and an alarm is issued if the verification fails. Compared with the offline data verification method in traditional practices, the present invention can perform high-quality verification on real-time distributed data, and while ensuring the verification quality, there is no need to overly rely on the quality and integrity of the use case. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The present invention is a flow chart of a distributed system data real-time verification method. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In a distributed system, since the functions of the entire system are implemented on various nodes, the data storage and processing of different nodes are also distributed. A complete business process running on a distributed system often calls multiple services at different stages, and each service may cause corresponding data source changes when it is called. With the increasing complexity of business processes and business environments, these distributed multi-data source changes are more prone to errors due to various reasons. Therefore, it is necessary to check and verify the changes of different data sources in order to find and correct errors in a timely manner.

[0026] The present invention provides a distributed system data real-time verification method, which is used to verify the data source changes caused by calling services in different links of a business process. In order to meet the implementation requirements of the distributed system data real-time verification method, two types of feature values ​​need to be pre-defined during the execution of the business process, the first type is the verification feature value, and the second type is the trigger feature value.

[0027] For the first type of verification feature values, each change in a data source when executing the same business process needs to be identified by writing the same verification feature value. In other words, the verification feature value is a unique identifier for data change records during the execution of successive business processes. For example, when a business process is executed, the service it calls needs to change data source A, data source B, and data source C. When changing these three data sources, each data change record needs to write a unique identification ID, that is, a verification feature value. The verification feature values ​​that need to be written to execute different business processes need to remain different. Therefore, all data change records corresponding to a certain business process can be extracted from all data sources based on the verification feature value in the future, so as to facilitate the verification of data changes caused by this business process.

[0028] For the second type of trigger characteristic value, it is necessary to pre-specify the execution status field of at least one key service in the business process as the trigger characteristic value. For example, when a business process is executed, the services it calls need to change data sources A, B, and C. Among all the called services, one service O1 is the most important and needs attention, and the execution status of the service O1 is recorded in the field L1 in the data source C. Therefore, field L1 can be used as a trigger characteristic value. If a value representing the execution of service O1 is written into field L1 in a row of data change records of data source C, it is considered that a trigger characteristic value has occurred. Of course, the specific selection of which key service needs to be made according to the design of the business data verification logic, which can be defined by yourself during the design process. Moreover, the trigger characteristic value can also be multiple fields, that is, it is considered that verification and verification need to be triggered only when multiple services are executed at the same time.

[0029] In a preferred embodiment of the present invention, after defining the above two types of characteristic values, a distributed system data real-time verification method can be implemented, and the real-time verification method includes the following process:

[0030] S1. The data sources to be checked involved in different services in the business process are respectively used as resource computing interfaces, and the messages of each resource computing interface are subscribed to. The data change records in each data source to be checked are obtained in real time, and a verification message is generated and sent to the parsing module as the data source of the parsing module.

[0031] It should be noted that the data source of the distributed system is generally stored on the node in the form of structured data, so the resource computing interface can be any type of structured data (such as a two-dimensional data table), depending on the actual data storage format in the system. Therefore, the specific form of the generated verification message will also be different according to the form of different structured data.

[0032] It should be noted that in a business process, not all changes in data sources require parameter verification. Which data sources need to be used as data sources to be verified can be designed based on the actual business data verification logic. Each data source to be verified is a separate resource calculation interface, through which data change records, i.e., verification messages, can be sent to the parsing module. In addition, there is a sequence of service execution in the business process, and the service execution time is also different, so the verification messages formed are sent to the parsing module asynchronously.

[0033] S2. After the parsing module receives the verification messages sent asynchronously by different resource computing interfaces, it calls the corresponding parsing rules according to the message type of each verification message, and parses the verification feature value and the trigger feature value from each verification message, and then sends the verification feature value in each verification message and the verification message itself as a key-value pair to the cache module for storage; at the same time, the parsing module adds the verification feature value and trigger feature value parsed from each verification message to the first trigger message and sends it to the trigger module.

[0034] It should be noted that the parsing rules are rules for parsing the target feature values ​​from the message information. Since different types of verification messages come from different data sources, their specific formats may be different, so the parsing rules required to parse the verification feature values ​​and trigger feature values ​​therefrom may also be different. In actual application, parsing rules can be pre-defined according to each type of verification message. After receiving the verification message, the corresponding parsing rules can be used according to its message type to parse the verification feature values ​​and trigger feature values ​​therefrom.

[0035] According to the above definitions of the check feature value and the trigger feature, the check feature value in the present invention is carried in each original check message, while the trigger feature value is unique to the check message corresponding to the pre-specified key service. Therefore, in some check messages, the trigger feature value may not exist, and the parsed value can be set to empty.

[0036] S3. After receiving the key-value pair data sent by the parsing module, the cache module stores them by category according to the key, and all the verification messages with the same key are merged to form a verification message set with the verification feature value as the key.

[0037] It should be noted that the main purpose of merging the verification messages using the verification feature value as a key in the present invention is to facilitate the subsequent verification of different data change records under the same execution operation process.

[0038] S4. After receiving the first trigger message sent by the parsing module, the trigger module calls the corresponding trigger rule according to the type of the trigger feature value, and adds the verification feature value in the first trigger message to the second trigger message and sends it to the verification module according to the trigger rule.

[0039] It should be noted that the triggering rules of the triggering module in the present invention support customization and can be adjusted according to the actual business logic design to meet the data requirements at the time of verification. As a preferred embodiment of the present invention, the triggering rules of the triggering module are set by default as follows: after receiving the first trigger message sent by the parsing module, a preset time is delayed, and after all data source changes caused by all services in a business process are executed and their respective verification messages are stored in the cache module, the verification feature value in the first trigger message is added to the second trigger message and sent to the verification module to trigger the verification.

[0040] It should be further explained that the "preset time" required for the above-mentioned delay can be determined according to the time it takes for different data sources to be verified to be cached in the cache module during the actual execution of the business process, so as to ensure that all data source changes caused by all called services in a business process are executed, and the verification messages generated by these data source changes are stored in the cache module.

[0041] S5. The verification module reads a verification message set with the verification feature value as the key from the cache module according to the verification feature value in the second trigger message, and performs structured processing on all the verification messages in the verification message set to form structured data for verification; then, according to the type of the verification feature value, the verification rule of the corresponding type in the rule module is called to verify the structured data; if the verification fails, an alarm message is sent to the alarm strategy module.

[0042] As a preferred method of an embodiment of the present invention, the verification module is pre-installed with structured processing methods corresponding to different types of verification messages. When the second trigger message is received, the corresponding structured processing method is called for each type of verification message in the verification message set, and the information to be verified is extracted and converted into structured data for storage.

[0043] It should be noted that the specific verification rules called are set according to the needs of the verification business, and are used to determine whether the data changes in different data sources comply with the business logic specifications. Because in some complex business scenarios, there may be many business processes that need to be verified, each type of business process that needs to be verified can be set up with a corresponding verification feature value category. Different categories of verification feature values ​​can be distinguished by setting identifiers, etc., and different types of verification feature values ​​can be pre-set with corresponding types of verification rules in the rule module to facilitate calls when actually conducting verification business.

[0044] S6. After receiving the alarm message, the alarm strategy module issues an alarm according to the preset alarm strategy.

[0045] As a preferred embodiment of the present invention, the rule module pre-sets corresponding checking rules for each type of trigger feature value, and the checking rules are implemented by writing configuration files.

[0046] As a preferred method of the embodiment of the present invention, the alarm strategy includes two methods: alarming the person in charge individually and alarming a group of people.

[0047] The distributed system data real-time verification method described in S1 to S6 above is applied to a specific embodiment to demonstrate its specific implementation.

[0048] Example

[0049] In this embodiment, a distributed system data real-time verification method is demonstrated based on the scenario of exchanging gift certificates for cash and depositing them into an electronic wallet. In the exchange business process involved in this scenario, different services will be called to perform gift certificate cancellation and cash deposit respectively. The cancellation data of the gift certificate is recorded in the gift table, and after the user uses the gift certificate, its cancellation record will be written into the gift table, and the cash amount corresponding to the gift certificate will be deposited into the user's electronic wallet, and the specific amount is recorded in the cash table. Both tables are structured data tables. Since the gift table and the cash table are stored on different nodes of the distributed system, and the execution time of the corresponding services is not completely consistent, the data in the two tables may be wrong. Since the issuance of cash is a relatively critical operation in this scenario, the data verification node in this scenario is that when a sum of cash is deposited into the user's electronic wallet, it is necessary to verify whether the corresponding gift certificate has been cancelled. If it has not been cancelled, an alarm reminder needs to be issued to the person in charge or other related personnel. In order to ensure the implementation of subsequent data verification, it is set that during the execution of a redemption business process, the data change records in the gift table and the cash table need to be written with a unique gift ID as a verification feature value, that is, the cancellation record in the gift table needs to record the gift ID, and the cash record deposited in the cash table due to the use of the gift voucher also needs to write the gift ID. At the same time, when cash is deposited due to gift voucher redemption, a redemption entry field L is set to be written in the gift table, and this field is used as a trigger feature value.

[0050] Based on the above application scenario, the real-time data verification method of the distributed system is as follows:

[0051] In the first step, the gift table and cash table involved in different services in the redemption business process are used as resource calculation interfaces, and the binlog1 and binlog2 messages of the gift table and cash table are subscribed to. The data change records in each data source to be checked are obtained in real time, and a check message (checkLog, the checkLog generated based on binlog1 and binlog2 are recorded as checkLog1 and checkLog2 respectively) is sent to the parsing module as the data source of the parsing module.

[0052] In the second step, after the parsing module receives the verification messages sent asynchronously by the gift table and the cash table, it calls the corresponding parsing rules according to the message type of each verification message, and parses the verification feature value (gift ID) and the trigger feature value (exchange entry field L) from each verification message, and then sends the gift ID in each verification message and the verification message itself as a key-value pair (IDA:checkLog) to the cache module for storage; at the same time, the parsing module adds the gift ID and the exchange entry field L parsed from each verification message to the first trigger message and sends it to the trigger module.

[0053] It should be noted that the gift ID is carried in every binlog1 message and binlog2 message, while the redemption entry field L is unique to the binlog2 message.

[0054] In the third step, after receiving the key-value pair data sent by the parsing module, the cache module stores them by key. All the check messages with the same key are merged to form a check message set (checkLogs) with the check feature value as the key. The keys of different check message sets are different. In this embodiment, the cache module finally appends the check messages with the same key into a message set (IDA: checkLogs->checkLogs=checkLogs.Append(newCheckLog).

[0055] In the fourth step, after the trigger module receives the first trigger message sent by the parsing module, it calls the corresponding trigger rule according to the type of the trigger feature value, and adds the gift ID in the first trigger message to the second trigger message according to the trigger rule and sends it to the verification module. In this embodiment, the trigger rule is to verify after a delay of two minutes, that is, after receiving the first trigger message, the second trigger message is sent to the verification module after a delay of two minutes, so as to trigger the verification after all the verification messages in the entire business process are stored in the cache module.

[0056] In the fifth step, the verification module reads a verification message set with the verification feature value as the key from the cache module according to the verification feature value in the second trigger message, and performs structured processing on all the verification messages in the verification message set to form structured data for verification, in which checkLog1 is parsed as a gift object and checkLog2 is parsed as a cash object. Then, according to the type of the verification feature value, the verification rule of the corresponding type in the rule module is called to verify the structured data. If the verification fails, an alarm message is sent to the alarm strategy module. In this embodiment, the verification rules set by writing a configuration file are: check whether the gift type of the gift object is cash, whether the gift has been consumed, and whether the gift quantity is consistent with the cash quantity of the cash table object.

[0057] Step 6: After receiving the alarm message, the alarm strategy module will issue an alarm according to the preset alarm strategy. The alarm strategy can be set to two ways: find the person in charge individually or create a group alarm with multiple people, which can be customized by the user.

[0058] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0059] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A distributed system data real-time verification method, characterized in that: It is used to check the data source changes caused by calling services at different stages of a business process. When executing the same business process, any change of a data source needs to be written with the same verification feature value for identification. At the same time, the execution status field of at least one key service in the business process is pre-specified as the trigger feature value. The real-time verification method comprises: S1. The data sources to be checked involved in different services in the business process are respectively used as resource computing interfaces, and the messages of each resource computing interface are subscribed to. The data change records in each data source to be checked are obtained in real time, and a verification message is generated and sent to the parsing module as the data source of the parsing module; S2. After receiving the verification messages sent asynchronously by different resource computing interfaces, the parsing module calls the corresponding parsing rules according to the message type of each verification message, and parses the verification feature value and the trigger feature value from each verification message, and then sends the verification feature value and the verification message itself in each verification message as a key-value pair to the cache module for storage; at the same time, the parsing module adds the verification feature value and the trigger feature value parsed from each verification message to the first trigger message and sends it to the trigger module; S3. After receiving the key-value pair data sent by the parsing module, the cache module stores them by category according to the key, and merges all the verification messages with the same key into a verification message set with the verification feature value as the key; S4. After receiving the first trigger message sent by the parsing module, the trigger module calls the corresponding trigger rule according to the type of the trigger feature value, and adds the verification feature value in the first trigger message to the second trigger message according to the trigger rule and sends it to the verification module; S5. The verification module reads a verification message set with the verification feature value as a key from the cache module according to the verification feature value in the second trigger message, performs structured processing on all the verification messages in the verification message set to form structured data for verification; then, according to the type of the verification feature value, the verification rule of the corresponding type in the rule module is called to verify the structured data, and if the verification is wrong, an alarm message is sent to the alarm strategy module; S6. After receiving the alarm message, the alarm strategy module issues an alarm according to the preset alarm strategy.

2. A distributed system data real-time verification method according to claim 1, characterized in that: In S1, the resource computing interface is any type of structured data.

3. A distributed system data real-time verification method according to claim 1, characterized in that: In S2, the verification characteristic value is carried in each original verification message, while the trigger characteristic value is unique to the verification message corresponding to the pre-specified key service.

4. A distributed system data real-time verification method according to claim 1, characterized in that: In S4, the triggering rules of the triggering module support customization.

5. A distributed system data real-time verification method according to claim 1, characterized in that: In S4, the triggering rule of the trigger module is: after receiving the first trigger message sent by the parsing module, delay for a preset time, wait until the data source changes caused by all services in a business process are executed and the respective verification messages are stored in the cache module, then add the verification feature value in the first trigger message to the second trigger message and send it to the verification module to trigger the verification.

6. A distributed system data real-time verification method according to claim 1, characterized in that: In S5, the verification module is pre-installed with structured processing methods corresponding to different verification message types. When the second trigger message is received, the corresponding structured processing method is called for each type of verification message in the verification message set, and the information to be verified is extracted and converted into structured data for storage.

7. A distributed system data real-time verification method according to claim 1, characterized in that: In S5, the rule module pre-sets corresponding checking rules for each type of trigger feature value, and the checking rules are implemented by writing a configuration file.

8. A distributed system data real-time verification method according to claim 1, characterized in that: In S6, the alarm strategy includes two methods: alarming the person in charge individually and alarming multiple people in a group.

Images