Data auditing method and device, electronic equipment and storage medium
By obtaining and analyzing the change notification of the database, generating incremental data and building incremental databases, the problem of inefficient data audits in the existing technology is solved, and more efficient and accurate data consistency detection is achieved.
Patent Information
- Application Number
- CN202411772402.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-02-18
AI Technical Summary
When the existing data auditing method handles high-concurrent incremental synchronization, it is difficult to completely avoid data inconsistencies, and consumes a large amount of computing resources and network resources, making it inefficient.
By obtaining the change notifications of the two databases, standardized analysis processing is performed, incremental data is generated, and incremental database is built based on these incremental data and historical incremental data to conduct data auditing.
It improves the efficiency of data auditing, reduces the consumption of computing resources and network resources, and can more accurately detect data consistency.
Smart Images
Figure CN119474075A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of data processing technology, and specifically relates to a data auditing method, device, electronic equipment and storage medium. Background Art
[0002] With the widespread application of technologies such as big data, cloud computing, and artificial intelligence, data has become a core resource for the development of enterprises and society. As the real-time, concurrency, and massive features of data processing become increasingly prominent, ensuring the consistency, integrity, and accuracy of data faces unprecedented challenges. Once data inconsistency occurs, it may not only lead to business logic errors and decision-making errors, but also cause serious compliance risks and user trust crises. Therefore, data auditing is the primary means to avoid this risk. The existing data auditing method using third-party tools consumes a lot of computing resources and network resources, and may still not be able to completely avoid data inconsistency when processing incremental data synchronization under high concurrency.
[0003] Therefore, a data auditing method is needed that can improve the efficiency of data auditing. Summary of the invention
[0004] The embodiment of the present application provides a data auditing method, which can improve the efficiency of data auditing.
[0005] In a first aspect, an embodiment of the present application provides a data audit method, the method comprising: obtaining a first change notification sent by a first database and a second change notification sent by a second database, the first change notification being used to characterize data change information of the first database, and the second change notification being used to characterize data change information of the second database; performing standardized parsing processing on the first change notification and the second change notification, respectively, to obtain first incremental data and second incremental data; determining a first incremental database based on the first incremental data and historical incremental data of the first database, and determining a second incremental database based on the second incremental data and historical incremental data of the second database; performing data audit processing based on the first incremental database and the second incremental database to obtain a data audit result.
[0006] In the second aspect, an embodiment of the present application provides a data auditing device, which includes: a first acquisition module, used to acquire a first change notification sent by a first database and a second change notification sent by a second database, the first change notification being used to characterize data change information of the first database, and the second change notification being used to characterize data change information of the second database; a first parsing module, used to perform standardized parsing processing on the first change notification and the second change notification, respectively, to obtain first incremental data and second incremental data; a first determination module, used to determine a first incremental database based on the first incremental data and historical incremental data of the first database, and to determine a second incremental database based on the second incremental data and historical incremental data of the second database; a first processing module, used to perform data audit processing based on the first incremental database and the second incremental database to obtain a data audit result.
[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.
[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
[0010] In an embodiment of the present application, a first change notification sent by a first database and a second change notification sent by a second database are obtained, the first change notification is used to represent the data change information of the first database, and the second change notification is used to represent the data change information of the second database; the first change notification and the second change notification are respectively subjected to standardized parsing processing to obtain first incremental data and second incremental data; a first incremental database is determined based on the first incremental data and the historical incremental data of the first database, and a second incremental database is determined based on the second incremental data and the historical incremental data of the second database; data audit processing is performed based on the first incremental database and the second incremental database to obtain data audit results, thereby improving data audit efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 It is a flowchart of a data audit method provided in an embodiment of the present application; Figure 2 It is a flowchart of the second data audit method provided in the embodiment of the present application; Figure 3 It is a flowchart of a third data audit method provided in an embodiment of the present application; Figure 4 It is a structural diagram of a data audit system provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of a data audit device provided in an embodiment of the present application; Figure 6 It is a structural diagram of a data auditing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0013] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0014] The data audit method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
[0015] Figure 1 A data audit method provided by an embodiment of the present invention is shown. The method can be executed by an electronic device, and the electronic device may include: a server and / or a terminal device, wherein the terminal device may be, for example, a vehicle-mounted terminal or a mobile phone terminal. In other words, the method can be executed by software or hardware installed in the data audit device, and the method includes the following steps: Step 102: Obtain a first change notification sent by the first database and a second change notification sent by the second database.
[0016] The first change notification is used to represent the data change information of the first database, and the second change notification is used to represent the data change information of the second database.
[0017] The client obtains a first change notification sent to itself by the first database and a second change notification sent to itself by the second database, wherein the first change notification is generated by the first database when performing a data change operation, and includes relevant information of the data change operation performed by the first database, and the second change notification is generated by the second database when performing a data change operation, and includes relevant information of the data change operation performed by the second database.
[0018] That is, when the first database and the second database perform data change operations, they will generate a first change notification and a second change notification based on their respective data change operations, respectively, to notify the client that the first database and the second database have performed data change operations, and to enable the client to understand the relevant information of the data change operations performed by the first database and the second database. After the first database and the second database send the first change notification and the second change notification to the client respectively, the client receives the first change notification and the second change notification sent by the first database and the second database.
[0019] Specifically, the first database and the second database may be Redis databases or other databases, and the data change operations performed in the first database and the second database may be data addition operations, data modification operations, and data deletion operations. An incremental data acquisition module may be installed in the client, and the incremental data acquisition module may receive the first change notification and the second change notification sent by the first database and the second database.
[0020] Step 104: Perform standardized parsing processing on the first change notification and the second change notification respectively to obtain first incremental data and second incremental data.
[0021] After receiving the first change notification and the second change notification sent by the first database and the second database respectively, the client performs standardized parsing on the first change notification to obtain the first incremental data corresponding thereto, and performs standardized parsing on the second change notification to obtain the second incremental data corresponding thereto. In other words, the message formats of the first change notification and the second change notification sent by the first database and the second database may be different. The client converts the first change notification and the second change notification into a unified format by performing standardized parsing on them, thereby facilitating data audit operations based on the first incremental data and the second incremental data.
[0022] Specifically, the incremental data acquisition module can include an incremental data processing submodule, which can analyze and process the first change notification and the second change notification, extract specific fields in the time notification, and then determine the operation type of the data change operation (such as adding, updating or deleting data, etc.), the key name involved and the corresponding new and old values, and then the incremental data processing submodule will standardize and encapsulate the parsed change information to obtain the first incremental data and the second incremental data. When encapsulating the parsed change information, a timestamp message["timestamp"] = int(time.time() * 1000) can also be added at the same time, so as to facilitate data audit processing based on the first incremental data and the second incremental data. By performing standardized parsing processing on the first change notification and the second change notification, the client can convert the information generated by the change operation of the first database and the second database into a standard form, so that no matter which database the change event comes from, the message object can follow the same structure and semantic conventions.
[0023] Step 106: Determine a first incremental database based on the first incremental data and the historical incremental data of the first database, and determine a second incremental database based on the second incremental data and the historical incremental data of the second database.
[0024] After obtaining the first incremental data and the second incremental data, the client needs to obtain the historical incremental data of the first database and the historical incremental data of the second database, and then determine the first incremental database based on the first incremental data and the historical incremental data of the first database, and determine the second incremental database based on the second incremental data and the historical incremental data of the second database, wherein the historical incremental data is obtained by the client performing standardized parsing on relevant information of historical change operations of the database, for example, the historical incremental data of the first database is determined based on relevant information of historical change operations of the first database, and the historical incremental data of the second database is determined based on relevant information of historical change operations of the second database.
[0025] Furthermore, whenever a data change operation is performed in the first database or the second database, it will send change information to the client so that the client will store the incremental data determined according to the change information, that is, the first incremental database stores all the latest incremental data of the first database, and the second incremental database stores all the latest incremental data of the second database.
[0026] Specifically, the incremental data acquisition module can also include an incremental data push submodule. After the incremental data processing submodule determines the first incremental data and the second incremental data, the incremental data push submodule sends the first incremental data and the second incremental data to the message subscription receiving submodule of the incremental data update comparison module installed in the client, and the message subscription receiving submodule forwards the first incremental data and the second incremental data to the message consumption submodule of the incremental data update comparison module, and then the message consumption submodule determines the first incremental database according to the first incremental data and the historical incremental data of the first database, and determines the second incremental database according to the second incremental data and the historical incremental data of the second database.
[0027] Step 108: Perform data audit processing based on the first incremental database and the second incremental database to obtain a data audit result.
[0028] After determining the first incremental database and the second incremental database, the client performs data audit processing based on the first incremental database and the second incremental database to obtain a data audit result, wherein the data audit result is used to characterize whether there is abnormal data in the first database and the second database.
[0029] Specifically, when performing data audit processing based on the first incremental database and the second incremental database, the client can determine the data audit result by determining whether the incremental data in the first incremental database is the same as the incremental data in the second incremental database. For example, when the incremental data in the first incremental database is the same as the incremental data in the second incremental database (that is, the number of incremental data in the first incremental database and the second incremental database is the same, and for any incremental data in the first incremental database, there is an identical incremental data in the second incremental database), it is determined that there is no abnormal data in the first database and the second database. If there is a difference, the different incremental data is determined to be abnormal data.
[0030] More specifically, the incremental data update comparison module can also include an audit comparison submodule, which can perform data audit processing based on the first incremental database and the second incremental database to obtain a data audit result.
[0031] The data audit method provided by the embodiment of the present invention obtains a first change notification sent by a first database and a second change notification sent by a second database, wherein the first change notification is used to characterize the data change information of the first database, and the second change notification is used to characterize the data change information of the second database; the first change notification and the second change notification are respectively subjected to standardized parsing processing to obtain first incremental data and second incremental data; a first incremental database is determined based on the first incremental data and the historical incremental data of the first database, and a second incremental database is determined based on the second incremental data and the historical incremental data of the second database; data audit processing is performed based on the first incremental database and the second incremental database to obtain a data audit result, thereby improving the efficiency of data auditing.
[0032] In one implementation, the obtaining of the first change notification sent by the first database and the second change notification sent by the second database (step 102) may include executing steps A1-A2: Step A2: Setting the first parameter of the first database and the second parameter of the second database to an on state, so that the first database and the second database respectively send the first change notification and the second change notification to the client when data is changed.
[0033] The client sets the first parameter of the first database and the second parameter of the second database to an on state, so that when the data in the first database and the second database change, the first database and the second database can send the first change notification and the second change notification to the client, wherein, when the first database and the second database are databases of the same type, such as a Redis database, the first parameter and the second parameter can be the same parameters of different databases.
[0034] Specifically, when both the first database and the second database are Redis databases, it is possible to ensure that the notify-keyspace-events parameter in the Redis configuration file redis.conf is enabled and set to notify-keyspace-events AKE, so that the client can receive notifications of database key changes in real time. Here, AKE is a combined identifier, where A represents the switch of all keyspace notification events, and K represents keyspace notification, that is, __keyspace@ <db>Events prefixed with __:, such as key creation and deletion, and E represents key expiration events.
[0035] Step A2: Initialize the first database connection pool and the second database connection pool.
[0036] The first database connection pool is used to asynchronously obtain the first change notification sent by the first database, and the second database connection pool is used to asynchronously obtain the second change notification sent by the second database. Furthermore, when the first database and the second database are databases of the same type, the client can also initialize a database connection pool and asynchronously obtain the change notifications sent by the first database and the second database through the database connection pool.
[0037] Specifically, the client initializes a Redis connection pool asynchronously by calling the create_redis_connection_pool function. This function creates a connection pool to the Redis server, which is used to manage and reuse the client's connection to Redis. After acquiring the Redis connection (through redis_pool.acquire()), the client creates a pubsub object (through redis.pubsub()), which is used to handle Redis's publish / subscribe function. Next, the client uses pubsub.psubscribe('__keyevent@*__:*') to subscribe to redis's key space events. After subscribing to the key space events, the client continuously calls pubsub.get_message() to asynchronously obtain event messages, i.e., change notifications, from the Redis server. Once a change notification is received, the client will perform the following operations: decode the message data (message['data'].decode('utf-8')) to obtain the specific content of the event; generate a UUID as the unique identifier of the event; call the process_event_with_key function to asynchronously process the event. This function first obtains the value of the key associated with the event from Redis (through the fetch_key_value function), and then prints out the relevant information of the event (including UUID, key and value). Furthermore, the incremental data collection module can also include an incremental data monitoring submodule, which can use the asynchronous IO model for asynchronous monitoring, effectively utilize system resources, improve the efficiency of data collection, and ensure that data changes will not be blocked or missed due to too many events in high concurrency scenarios.
[0038] In one implementation, the determining of the first incremental database based on the first incremental data and the historical incremental data of the first database, and the determining of the second incremental database based on the second incremental data and the historical incremental data of the second database (step 106) may include executing steps B1-B5: Step B1: input the first incremental data and the second incremental data into the first message queue and the second message queue respectively.
[0039] The first message queue is used to temporarily store the first incremental data, and the second message queue is used to temporarily store the second incremental data.
[0040] After obtaining the first incremental data and the second incremental data, the client can input the first incremental data into the first message queue and the second incremental data into the second message queue, wherein the first message queue is used to temporarily store the first incremental data and the second message queue is used to temporarily store the second incremental data.
[0041] Specifically, after the encapsulated standardized message object (i.e., the first incremental data and the second incremental data) is submitted to the incremental data push sub-module, the sub-module pushes the first incremental data to the pre-set first message queue and the second message queue. The message queue can be kafak, and the Kafka producer instance is pre-configured before this, including specifying the Kafka server address, setting the serialization method, etc.
[0042] As middleware, the first and second message queues play the role of temporary storage, buffering and distribution, ensuring that data changes can be delivered to subsequent audit processing links in an orderly and timely manner in a high-concurrency environment to avoid direct impact on the performance of the core business system. At the same time, the message queue also supports horizontal expansion and can dynamically adjust its processing capacity according to actual needs to cope with the growing audit needs and data volume.
[0043] Step B2: After obtaining the first incremental data from the first message queue, determine a first processing partition for processing the first incremental data, and a first consumer group corresponding to the first processing partition.
[0044] After sending the first incremental data to the first message queue, the client obtains the first incremental data from the first message queue, and determines the first processing partition for processing the first incremental data, and the first consumer group corresponding to the first processing partition. In other words, in order to improve consumption performance, the client logically partitions the message queue, uses multiple threads or multiple processes for consumption of a single partition, and each partition is consumed by an independent consumer thread or process, so that messages can be processed in parallel and throughput can be improved. By implementing this efficient and reliable audit data update mechanism, the system can receive and process change events in the data source in real time and accurately, ensuring the integrity and consistency of incremental data.
[0045] Specifically, when determining the first processing partition according to the first incremental data, the first processing partition for processing the first incremental data can be determined according to the change operation type (such as addition, deletion or modification) of the first incremental data. In other words, different types of first incremental data are processed using different processing partitions, and each processing partition has a corresponding consumer group. Furthermore, the message subscription receiving submodule can subscribe to the change event topic transmitted by the data source acquisition module in real time. According to the preset subscription rules, the module intelligently filters and distributes change events of different topics to ensure that each change event can be processed by the correct consumer. For each change event received, the module can also use a serial number or version number for unique identification and tracking to ensure the integrity and traceability of data changes.
[0046] Step B3: Update the historical incremental data of the first database through the first consumer group to determine the first incremental database.
[0047] After determining the first processing partition for processing the first incremental data and the first consumer group corresponding to the first processing partition, the client updates the historical incremental data of the first database through the first consumer group, thereby determining the first incremental database. Specifically, after receiving the change event, the message consumption submodule executes the corresponding data processing logic according to the event type and content. After processing the change event, the module will update the incremental data and determine the first incremental database to ensure the accuracy and consistency of the data. At the same time, the module can also record the log information of the update operation for subsequent auditing and tracing.
[0048] Step B4: After obtaining the second incremental data from the second message queue, determine a second processing partition for processing the second incremental data, and a second consumer group corresponding to the second processing partition.
[0049] After sending the second incremental data to the second message queue, the client obtains the second incremental data from the first message queue, and determines the second processing partition for processing the second incremental data, and the second consumer group corresponding to the second processing partition. The specific implementation method is the same as the above-mentioned execution method of determining the first processing partition and determining the first consumer group, which will not be repeated here.
[0050] Step B4: Update the historical incremental data of the second database through the second consumer group to determine the second incremental database.
[0051] After determining the second processing partition for processing the second incremental data and the second consumer group corresponding to the second processing partition, the client updates the historical incremental data of the second database through the second consumer group, thereby determining the second incremental database. The specific implementation method is the same as the above-mentioned updating of the historical incremental data of the first database through the first consumer group to determine the first incremental database, and will not be repeated here.
[0052] Figure 2 is a flow chart of a second data audit method provided in an embodiment of this specification, such as Figure 2 As shown, the schematic diagram includes: Step 202: Setting a first parameter of the first database and a second parameter of the second database to an on state, so that the first database and the second database send a first change notification and a second change notification to the client, respectively, when data is changed.
[0053] The first change notification is used to represent the data change information of the first database, and the second change notification is used to represent the data change information of the second database.
[0054] Step 204: Initialize the first database connection pool and the second database connection pool.
[0055] The first database connection pool is used to asynchronously obtain the first change notification sent by the first database, and the second database connection pool is used to asynchronously obtain the second change notification sent by the second database.
[0056] Step 206: Obtain the first change notification sent by the first database and the second change notification sent by the second database.
[0057] Step 208: Perform standardized parsing processing on the first change notification and the second change notification respectively to obtain first incremental data and second incremental data.
[0058] Step 210: Input the first incremental data and the second incremental data into the first message queue and the second message queue respectively.
[0059] The first message queue is used to temporarily store the first incremental data, and the second message queue is used to temporarily store the second incremental data.
[0060] Step 212: After obtaining the first incremental data from the first message queue, determine a first processing partition for processing the first incremental data, and a first consumer group corresponding to the first processing partition.
[0061] Step 214: Update the historical incremental data of the first database through the first consumer group to determine the first incremental database.
[0062] Step 216: After obtaining the second incremental data from the second message queue, determine a second processing partition for processing the second incremental data, and a second consumer group corresponding to the second processing partition.
[0063] Step 218: Update the historical incremental data of the second database through the second consumer group to determine the second incremental database.
[0064] Step 220: Perform data audit processing based on the first incremental database and the second incremental database to obtain a data audit result.
[0065] In the embodiments of the specification, by asynchronously acquiring the first change notification and the second change notification in the first database and the second database, it is possible to effectively utilize system resources, improve the efficiency of data collection, and ensure that data changes will not be blocked or missed due to too many notification events in high-concurrency scenarios; by inputting the first incremental data and the second incremental data into the first message queue and the second message queue for temporary storage, it is ensured that in a high-concurrency environment, the data can be delivered to the subsequent audit processing links in an orderly and timely manner, thereby avoiding direct impact on the performance of the core business system.
[0066] In one implementation, the data audit process is performed based on the first incremental database and the second incremental database to obtain a data audit result (step 108), and steps C1-C2 may be performed: Step C1: Determine whether the first incremental database includes the second incremental data, and determine whether the second incremental database includes the first incremental data.
[0067] After determining the first incremental database and the second incremental data, the client determines whether the first incremental database includes the second incremental data; after determining the first incremental data and the second incremental database, the client determines whether the second incremental database includes the first incremental data.
[0068] Step C2: If it is determined that the first incremental database includes the second incremental data, and the second incremental database includes the first incremental data, then it is determined that the data audit result is no abnormal data.
[0069] When the client performs data audit processing based on the first incremental database and the second incremental database, the client can not only obtain the data audit result by determining whether the first incremental database and the second incremental database are the same, but also can determine whether the first incremental database includes the second incremental data and whether the second incremental database includes the first incremental data. That is, when it is determined that the first incremental database includes the second incremental data and the second incremental database includes the first incremental data, it is determined that the data audit result is no abnormal data.
[0070] Specifically, the client will perform a data audit in real time after receiving the first incremental data and the second incremental data each time, and determine the data audit result. Then, when receiving new first incremental data and second incremental data, the client can determine the data audit result by determining whether the first incremental database includes the newly received second incremental data and whether the second incremental database includes the newly received first incremental data. In other words, when determining that there is no abnormal data in the historical incremental data in the first incremental database and the second incremental database, the client can determine that the data audit result is no abnormal data by determining that the first incremental database includes the second incremental data and the second incremental database includes the first incremental data. More specifically, the client can also determine the data audit result by determining whether the first incremental data and the second incremental data are the same, that is, when the first incremental data and the second incremental data are the same, the data audit result is determined to be no abnormal data, and when the first incremental data and the second incremental data are not the same, the data audit result is determined to be the presence of abnormal data.
[0071] In one implementation, the incremental data stored in the first incremental database and the second incremental database are classified and stored according to preset types, and the data auditing process is performed based on the first incremental database and the second incremental database to obtain the data auditing result (step 108), and steps D1-D6 may be executed: Step D1: Acquire the third incremental data in the first incremental database, and determine the first type of the third incremental data.
[0072] The third incremental data is any incremental data in the first incremental database.
[0073] When storing the incremental data in the first incremental database and the second incremental database, the client can classify and store the incremental data, for example, by operation type, for example, by operation object.
[0074] The client obtains the third incremental data in the first incremental database, and determines the type of the third incremental data, that is, the first type, wherein the third incremental data is any incremental data in the first incremental database. In other words, the client selects any incremental data from the first incremental database as the third incremental data, and determines the type of the incremental data.
[0075] Specifically, when the first incremental database and the second incremental database store incremental data according to operation type, the incremental data can be classified and stored according to added incremental data, updated incremental data and deleted incremental data; when the first incremental database and the second incremental database are classified according to operation objects, the client content can maintain a mapping table based on key-value pairs, so as to store different incremental data of the same operation object.
[0076] Step D2: Determine whether there is incremental data identical to the third incremental data in the incremental data corresponding to the first type in the second incremental database.
[0077] After determining the third incremental data and the first type in the first incremental database, the client determines whether there is incremental data identical to the third incremental data among the plurality of incremental data corresponding to the first type in the second incremental database. In other words, the client determines the incremental data of the first type stored in the second incremental database, and then determines whether the incremental data of the first type stored therein includes the third incremental data.
[0078] Specifically, when the incremental data in the first incremental database and the second incremental database are classified and stored according to the operation type, the incremental data corresponding to the operation type of the third incremental data in the second incremental database can be compared with the third incremental data to determine whether there is incremental data identical to the third incremental data. When the incremental data in the first incremental database and the second incremental database are classified and stored according to the operation object, the incremental data in the second incremental database that has the same operation object as the third incremental data can be compared with the third type of incremental data to determine whether there is incremental data identical to the third incremental data.
[0079] Step D3: If not, then determine that the data audit result is that abnormal data exists.
[0080] If the client determines that the incremental data of the first type stored in the second incremental database does not include the third incremental data, then the data audit result is determined to be abnormal data. If the client determines that the incremental data of the first type stored in the second incremental database includes the third incremental data, that is, any incremental data in the first incremental database can be determined to be the same as the data in the first incremental database in the incremental data corresponding to its type in the second incremental database, then the data audit result is determined to be no abnormal data.
[0081] That is to say, when any incremental data in the first incremental database can be identified as the same incremental data in the incremental data of the same type as the incremental data in the second incremental database, then the data audit result can be determined to be that there is no abnormal data. When there is any incremental data that cannot be identified as the same incremental data in the incremental data of the same type as the incremental data in the second incremental database, then the data audit result can be determined to be that there is abnormal data.
[0082] Step D4: Obtain fourth incremental data in the second incremental database, and determine the second type of the fourth incremental data.
[0083] The fourth incremental data is any incremental data in the second incremental database.
[0084] The client obtains the fourth incremental data in the second incremental database, and determines the type of the fourth incremental data, that is, the second type, wherein the fourth incremental data is any incremental data in the second incremental database. In other words, the client selects any incremental data from the second incremental database as the fourth incremental data, and determines the type of the incremental data.
[0085] Specifically, the client obtains the fourth incremental data in the second incremental database and determines the second type of implementation method of the fourth incremental data, which is the same as the above-mentioned method of obtaining the third incremental data in the first incremental database and determining the first type of implementation method of the third incremental data, and will not be repeated here.
[0086] Step D5: Determine whether there is incremental data identical to the fourth incremental data in the incremental data corresponding to the second type in the first incremental database.
[0087] After determining the fourth incremental data and the second type in the second incremental database, the client determines whether there is incremental data identical to the fourth incremental data among the plurality of incremental data corresponding to the second type in the first incremental database. In other words, the client determines the incremental data of the second type stored in the first incremental database, and then determines whether the fourth incremental data is included in the incremental data of the second type stored therein.
[0088] Specifically, the client determines whether there is an implementation method of incremental data identical to the fourth incremental data in the incremental database corresponding to the second type in the first incremental database. This is the same as the above-mentioned method of determining whether there is an implementation method of incremental data identical to the third incremental data in the incremental database corresponding to the first type in the second incremental database, and will not be repeated here.
[0089] Step D6: If not, then determine that the data audit result is that abnormal data exists.
[0090] If the client determines that the second type of incremental data stored in the first incremental database does not include the fourth incremental data, then the data audit result is determined to be abnormal data. If the client determines that the second type of incremental data stored in the first incremental database includes the fourth incremental data, that is, any incremental data in the second incremental database can be determined to be the same as the data in the first incremental database in the incremental data corresponding to its type in the first incremental database, then the data audit result is determined to be no abnormal data.
[0091] Figure 3 is a flow chart of a third data audit method provided in an embodiment of this specification, such as Figure 3 As shown, the schematic diagram includes: Step 302: Obtain a first change notification sent by the first database and a second change notification sent by the second database.
[0092] The first change notification is used to represent the data change information of the first database, and the second change notification is used to represent the data change information of the second database.
[0093] Step 304: Perform standardized parsing processing on the first change notification and the second change notification respectively to obtain first incremental data and second incremental data.
[0094] Step 306: Determine a first incremental database based on the first incremental data and the historical incremental data of the first database, and determine a second incremental database based on the second incremental data and the historical incremental data of the second database.
[0095] Step 308: Obtain third incremental data in the first incremental database, and determine the first type of the third incremental data.
[0096] The third incremental data is any incremental data in the first incremental database.
[0097] Step 310: Determine whether there is incremental data identical to the third incremental data in the incremental data corresponding to the first type in the second incremental database.
[0098] Step 312: If not, determine that the data audit result is that abnormal data exists.
[0099] Step 314: Obtain fourth incremental data in the second incremental database, and determine the second type of the fourth incremental data.
[0100] The fourth incremental data is any incremental data in the second incremental database.
[0101] Step 316: Determine whether there is incremental data identical to the fourth incremental data in the incremental data corresponding to the second type in the first incremental database.
[0102] Step 318: If not, determine that the data audit result is that abnormal data exists.
[0103] In the embodiments of the specification, by classifying and storing the incremental data in the first incremental database and the second incremental database according to preset types, when performing data auditing, auditing is performed according to the type of data to be audited, thereby reducing the number of data that needs to be compared during the auditing process and improving the efficiency of data auditing.
[0104] In one implementation, steps E1-E2 may also be performed: Step E1: Send a heartbeat instruction to the first database and the second database, so that the first database and the second database send a heartbeat feedback to the client.
[0105] The client can send a heartbeat instruction to the first database so that the first database can feed back heartbeat feedback to the client, and send a heartbeat instruction to the second database so that the second database can feed back heartbeat feedback to the client, wherein the heartbeat instruction is used to instruct the first database and the second database to send heartbeat feedback to the client at a preset time interval, and the heartbeat feedback is used to characterize the connection status of the first database or the second database with the client.
[0106] Specifically, an audit and exception handling module may be installed in the client, which may include an exception detection submodule for real-time monitoring of the network and operating status. A heartbeat detection timer task is pre-set in the exception detection submodule, so that a heartbeat packet (i.e., a heartbeat instruction) is sent to the first database and the second database at each predetermined time interval, so that the first database and the second database send feedback information, i.e., heartbeat feedback, to the client when the network is normal.
[0107] Step E2: When the heartbeat feedback sent by the first database and / or the second database is not received within a preset time interval, it is determined that an abnormality has occurred in the first database and / or the second database, and the data audit operation on the first incremental database and the second incremental database is stopped.
[0108] When the client receives heartbeat feedback sent by the first database and the second database, the client can determine that the connection with the first database and the second database is normal. When the client does not receive heartbeat feedback sent by the first database and / or the second database within a preset time interval, the client can determine that an abnormality has occurred between it and the first database and / or the second database, and then stop the data audit operation.
[0109] Specifically, the audit and exception handling module may also include an exception handling submodule. When the monitoring mechanism deployed inside the exception detection submodule detects that the heartbeat feedback sent by the first database and / or the second database is not received within the preset time interval, it will be considered that the connection with the first database and / or the second database is abnormal, and the heartbeat detection timeout notification will be triggered. After the exception handling submodule receives the heartbeat detection timeout notification, the exception handling submodule will immediately trigger the automatic pause mechanism to suspend the incremental comparison, thereby avoiding the potential risks of data in an unstable network environment. At the same time, the module will quickly capture and record the status information of the current change tracking, including key data such as the data version number being processed and the transaction sequence number. Change events affected by network fluctuations will be temporarily stored in a memory buffer or a disk file. This module creates a first-in-first-out (FIFO) queue or a ring buffer in the memory for temporarily storing change events, ensuring that data change events are safely stored during network instability. After the network is restored, the system can continue processing from the breakpoint to ensure data integrity and continuity of the processing flow.
[0110] Figure 4 is a structural diagram of a data audit system provided in an embodiment of this specification, such as Figure 4 As shown, the system includes: an incremental data acquisition module, an incremental data update comparison module and an audit and exception handling module, wherein the incremental data acquisition module includes an incremental data monitoring submodule, an incremental data processing submodule and an incremental data push submodule, the incremental data update comparison module includes a message subscription receiving submodule, a message consumption submodule and an audit comparison submodule, and the audit and exception handling module includes an exception detection submodule and an exception handling submodule. The incremental data monitoring submodule monitors the change events of the first database and the second database, and sends the monitored first change notification and the second change notification to the incremental data processing submodule. The incremental data processing submodule performs parsing based on the first change notification and the second change notification, and sends the parsed first incremental data and the second incremental data to the incremental data pushing submodule, so that it pushes the first incremental data and the second incremental data to the message subscription receiving submodule. The message subscription receiving submodule forwards the first incremental data and the second incremental data to the message consumption submodule, and the message consumption submodule determines the first incremental database based on the first incremental data and determines the second incremental database based on the second incremental data, so that the audit comparison submodule performs data audit based on the first incremental database and the second incremental database. The abnormality detection submodule can send a heartbeat instruction to the first database and the second database, so that the first database and the second database send a heartbeat feedback to the client, and then determine the connection status according to the feedback result. The abnormality handling submodule can stop the audit operation on the first incremental database and the second incremental database when the connection between the first database and the second database is abnormal. It should be noted that the data audit method provided in the embodiment of the present application can be executed by a data audit device or a control module in the data audit device for executing the data audit method. In the embodiment of the present application, the data audit device provided in the embodiment of the present application is described by taking the data audit device executing the data audit method as an example.
[0111] Figure 5 Schematic diagram of the structure of a data audit device according to an embodiment of the present invention. Figure 5 As shown, the data auditing device includes: a first acquisition module 502 , a first analysis module 504 , a first determination module 506 , and a first processing module 508 .
[0112] A first acquisition module 502 is used to acquire a first change notification sent by a first database and a second change notification sent by a second database, wherein the first change notification is used to represent data change information of the first database, and the second change notification is used to represent data change information of the second database; A first parsing module 504 is used to perform standardized parsing processing on the first change notification and the second change notification respectively to obtain first incremental data and second incremental data; A first determination module 506, configured to determine a first incremental database based on the first incremental data and historical incremental data of the first database, and to determine a second incremental database based on the second incremental data and historical incremental data of the second database; The first processing module 508 is used to perform data audit processing based on the first incremental database and the second incremental database to obtain a data audit result.
[0113] The data audit device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0114] The data audit device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0115] The data auditing device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented in the method embodiment are not described here.
[0116] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to execute the above-mentioned data audit method. Figure 6 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 602, a communication interface (Communications Interface) 604, a memory (memory) 606, and a communication bus 608, wherein the processor 602, the communication interface 604, and the memory 606 communicate with each other through the communication bus 608. The processor 602 may call a computer program stored in the memory 606 and executable on the processor 602 to perform the following steps: Acquire a first change notification sent by a first database and a second change notification sent by a second database, wherein the first change notification is used to represent data change information of the first database, and the second change notification is used to represent data change information of the second database; Performing standardized parsing processing on the first change notification and the second change notification respectively to obtain first incremental data and second incremental data; Determine a first incremental database based on the first incremental data and historical incremental data of the first database, and determine a second incremental database based on the second incremental data and historical incremental data of the second database; Data audit processing is performed based on the first incremental database and the second incremental database to obtain a data audit result.
[0117] In one implementation, obtaining a first change notification sent by the first database and a second change notification sent by the second database includes: Setting a first parameter of the first database and a second parameter of the second database to an on state, so that the first database and the second database send the first change notification and the second change notification to the client, respectively, when data changes; Initialize a first database connection pool and a second database connection pool, the first database connection pool is used to asynchronously obtain the first change notification sent by the first database, and the second database connection pool is used to asynchronously obtain the second change notification sent by the second database.
[0118] In one implementation, determining a first incremental database based on the first incremental data and historical incremental data of the first database, and determining a second incremental database based on the second incremental data and historical incremental data of the second database, includes: Inputting the first incremental data and the second incremental data into a first message queue and a second message queue respectively, wherein the first message queue is used for temporarily storing the first incremental data, and the second message queue is used for temporarily storing the second incremental data; After acquiring the first incremental data from the first message queue, determining a first processing partition for processing the first incremental data and a first consumer group corresponding to the first processing partition; Updating the historical incremental data of the first database by the first consumer group to determine the first incremental database; After acquiring the second incremental data from the second message queue, determining a second processing partition for processing the second incremental data, and a second consumer group corresponding to the second processing partition; The second incremental database is determined by updating the historical incremental data of the second database through the second consumer group.
[0119] In one implementation, performing data audit processing based on the first incremental database and the second incremental database to obtain a data audit result includes: Determine whether the first incremental database includes the second incremental data, and determine whether the second incremental database includes the first incremental data; If it is determined that the first incremental database includes the second incremental data, and the second incremental database includes the first incremental data, then it is determined that the data audit result is no abnormal data.
[0120] In one implementation, the incremental data stored in the first incremental database and the second incremental database are classified and stored according to preset types, and the data auditing process is performed based on the first incremental database and the second incremental database to obtain the data auditing result, including: Acquire third incremental data in the first incremental database, and determine a first type of the third incremental data, where the third incremental data is any incremental data in the first incremental database; Determine whether there is incremental data identical to the third incremental data in the incremental data corresponding to the first type in the second incremental database; If not, it is determined that the data audit result is that abnormal data exists; Acquire fourth incremental data in the second incremental database, and determine a second type of the fourth incremental data, where the fourth incremental data is any incremental data in the second incremental database; Determine whether there is incremental data identical to the fourth incremental data in the incremental data corresponding to the second type in the first incremental database; If not, it is determined that the data audit result is that abnormal data exists.
[0121] In one implementation, the method further includes: Sending a heartbeat instruction to the first database and the second database so that the first database and the second database send a heartbeat feedback to the client; When the heartbeat feedback sent by the first database and / or the second database is not received within a preset time interval, it is determined that an abnormality has occurred in the first database and / or the second database, and the data audit operation on the first incremental database and the second incremental database is stopped.
[0122] The specific execution steps can refer to the various steps of the above-mentioned data audit method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.
[0123] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.
[0124] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
[0125] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).
[0126] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.
[0127] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned data audit method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0128] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0129] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data audit method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0130] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0131] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0133] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.< / db>
Claims
1. A data audit method, applied to a client, comprising: Acquire a first change notification sent by a first database and a second change notification sent by a second database, wherein the first change notification is used to represent data change information of the first database, and the second change notification is used to represent data change information of the second database; Performing standardized parsing processing on the first change notification and the second change notification respectively to obtain first incremental data and second incremental data; Determine a first incremental database based on the first incremental data and historical incremental data of the first database, and determine a second incremental database based on the second incremental data and historical incremental data of the second database; Data audit processing is performed based on the first incremental database and the second incremental database to obtain a data audit result.
2. According to the method of claim 1, the obtaining of the first change notification sent by the first database and the second change notification sent by the second database comprises: Setting a first parameter of the first database and a second parameter of the second database to an on state, so that the first database and the second database send the first change notification and the second change notification to the client, respectively, when data changes; Initialize a first database connection pool and a second database connection pool, the first database connection pool is used to asynchronously obtain the first change notification sent by the first database, and the second database connection pool is used to asynchronously obtain the second change notification sent by the second database.
3. The method according to claim 1, wherein determining a first incremental database based on the first incremental data and historical incremental data of the first database, and determining a second incremental database based on the second incremental data and historical incremental data of the second database, comprises: Inputting the first incremental data and the second incremental data into a first message queue and a second message queue respectively, wherein the first message queue is used for temporarily storing the first incremental data, and the second message queue is used for temporarily storing the second incremental data; After acquiring the first incremental data from the first message queue, determining a first processing partition for processing the first incremental data and a first consumer group corresponding to the first processing partition; Updating the historical incremental data of the first database by the first consumer group to determine the first incremental database; After acquiring the second incremental data from the second message queue, determining a second processing partition for processing the second incremental data, and a second consumer group corresponding to the second processing partition; The second incremental database is determined by updating the historical incremental data of the second database through the second consumer group.
4. According to the method of claim 1, the step of performing data audit processing based on the first incremental database and the second incremental database to obtain a data audit result comprises: Determine whether the first incremental database includes the second incremental data, and determine whether the second incremental database includes the first incremental data; If it is determined that the first incremental database includes the second incremental data, and the second incremental database includes the first incremental data, then it is determined that the data audit result is no abnormal data.
5. The method according to claim 1, wherein the incremental data stored in the first incremental database and the second incremental database are classified and stored according to preset types, and the data auditing process is performed based on the first incremental database and the second incremental database to obtain the data auditing result, comprising: Acquire third incremental data in the first incremental database, and determine a first type of the third incremental data, where the third incremental data is any incremental data in the first incremental database; Determine whether there is incremental data identical to the third incremental data in the incremental data corresponding to the first type in the second incremental database; If not, then determining that the data audit result is that abnormal data exists; Acquire fourth incremental data in the second incremental database, and determine a second type of the fourth incremental data, where the fourth incremental data is any incremental data in the second incremental database; Determine whether there is incremental data identical to the fourth incremental data in the incremental data corresponding to the second type in the first incremental database; If not, it is determined that the data audit result is that abnormal data exists.
6. The method according to claim 1, further comprising: Sending a heartbeat instruction to the first database and the second database so that the first database and the second database send a heartbeat feedback to the client; When the heartbeat feedback sent by the first database and / or the second database is not received within a preset time interval, it is determined that an abnormality has occurred in the first database and / or the second database, and the data audit operation on the first incremental database and the second incremental database is stopped.
7. A data auditing device, comprising: A first acquisition module, configured to acquire a first change notification sent by a first database and a second change notification sent by a second database, wherein the first change notification is used to represent data change information of the first database, and the second change notification is used to represent data change information of the second database; A first parsing module, configured to perform standardized parsing processing on the first change notification and the second change notification respectively to obtain first incremental data and second incremental data; A first determination module, configured to determine a first incremental database based on the first incremental data and historical incremental data of the first database, and to determine a second incremental database based on the second incremental data and historical incremental data of the second database; The first processing module is used to perform data audit processing based on the first incremental database and the second incremental database to obtain a data audit result.
8. A computer device, characterized in that: The device comprises: Processor; and A memory arranged to store computer executable instructions, wherein the executable instructions are configured to be executed by the processor, and the executable instructions include instructions for executing the steps in the data audit method according to any one of claims 1 to 6.
9. A storage medium, characterized in that: The storage medium is used to store computer-executable instructions, and the executable instructions enable a computer to execute the data audit method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the data audit method described in any one of claims 1 to 6.