Data reporting method, data reporting toolkit and data auditing platform

Abstract

The application relates to the technical field of data processing, in particular to a data reporting method, a data reporting tool package and a data audit platform. The method comprises the following steps: in response to an external client calling a preset timer interface, audit data is collected from a preset database by using the timer interface; according to a preset rule, the audit data is pre-aggregated according to a preset field to obtain aggregated data; based on a lock-free design mode, the aggregated data is secondarily aggregated according to a preset data structure to obtain audit messages, and the audit messages are cached in a preset cache middleware; the audit messages in the cache middleware are written into a preset distributed message middleware in an asynchronous mode; and the distributed message middleware is used for asynchronously consuming the audit data by an external data audit system. Therefore, the problems of low performance, high coupling and poor flexibility of an existing data audit system can be effectively solved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data reporting method, a data reporting toolkit, and a data auditing platform. Background Technology

[0002] In real-time data computing scenarios, it is necessary to audit indicators such as the latency of each branch flow from end to end of the entire link, the overall latency of the link, and the health status of the link, in order to verify the accuracy and timeliness of the data and monitor the operational status of the link.

[0003] However, most monitoring and data auditing systems in the industry lack support for end-to-end auditing functions. For example, the reported indicators are relatively simple, there is a lack of support for custom indicators, and there are problems such as low performance, high coupling, and poor flexibility. Summary of the Invention

[0004] In view of this, embodiments of this application provide a data reporting method, a data reporting toolkit, and a data auditing platform, which can effectively solve the problems of low performance, high coupling, and poor flexibility of existing data auditing systems.

[0005] In a first aspect, embodiments of this application provide a data reporting method, the method comprising:

[0006] In response to an external client calling a preset timer interface, audit data is collected from a preset database using the timer interface;

[0007] According to preset rules, the audit data is pre-aggregated based on preset fields to obtain aggregated data;

[0008] Based on a lock-free design, the aggregated data is subjected to secondary aggregation processing according to a preset data structure to obtain an audit message, which is then cached in a preset cache middleware.

[0009] The audit message in the cache middleware is written to the preset distributed message middleware in an asynchronous manner;

[0010] The distributed message middleware is used to allow external data auditing systems to asynchronously consume the audit data.

[0011] In some embodiments, the audit data includes preset audit fields such as audit data type, data source table, and data source database;

[0012] The step of pre-aggregating the audit data according to preset rules and preset fields to obtain aggregated data includes:

[0013] Based on a preset time interval, a pre-aggregation key is determined for each piece of audit data according to preset fields;

[0014] For each of the pre-aggregation bonds, a unique corresponding atomic variable object is determined;

[0015] Aggregate all audit data entries with the same pre-aggregation key, and assign the resulting aggregated data to the corresponding atomic variable object.

[0016] In some embodiments, determining a unique corresponding atomic variable object for each of the pre-aggregation bonds includes:

[0017] Check if the pre-aggregation key has a corresponding atomic variable object; if it does not exist, generate a corresponding atomic variable object; if it exists, obtain the atomic variable object.

[0018] In some embodiments, the secondary aggregation process includes packet assembly and serialization; the secondary aggregation process, based on a lock-free design and a preset data structure, to obtain an audit message includes:

[0019] Based on the preset data structure, the corresponding atomic variable values ​​are packaged and serialized in batches at regular intervals through each atomic variable object to obtain the serialized data corresponding to each pre-aggregation key, which serves as the audit message corresponding to each pre-aggregation key;

[0020] After the grouping, the values ​​of each atomic variable are initialized.

[0021] In some embodiments, the step of periodically batch-packing and serializing the corresponding atomic variable values ​​through each atomic variable object based on the preset data structure to obtain serialized data corresponding to each pre-aggregation key includes:

[0022] According to the preset data structure, each atomic variable value is periodically and batch-combined into a corresponding protocol buffer object;

[0023] Based on a preset time interval, the corresponding first producer is obtained according to the audit data type of the audit message. The serialization method built into the protocol buffer is called through the first producer, and each protocol buffer object is serialized through the serialization method.

[0024] In some embodiments, the caching middleware further includes a first queue;

[0025] The caching to a preset caching middleware includes:

[0026] The first producer for each audit message type caches the corresponding audit message to the first queue.

[0027] In some embodiments, the caching middleware includes a first consumer; the distributed message middleware includes a producer pool, a second queue, and a second consumer.

[0028] The step of asynchronously writing the audit message from the cache middleware into a preset distributed message middleware includes:

[0029] After consuming the audit message event in the first queue, the first consumer obtains the corresponding second producer instance from the pre-built producer pool according to the message event label; and uses the asynchronous interface of the second producer instance to obtain the audit message and store it in the second queue.

[0030] In some embodiments, the method further includes one or more of the following three items:

[0031] First item: Based on at least one preset audit message type, generate a second producer instance corresponding to each audit message type in the producer pool using the eager initialization method;

[0032] The second item: If an error is detected during the process of sending the audit message to the second queue, an error log is recorded, and the audit message is retried asynchronously until the preset retry limit is reached; wherein, the audit message is guaranteed to be written with semantics at least once;

[0033] The third item: The preset data structure includes a first message class and a second message class; both the first message class and the second message class include multiple indicator fields; the protocol buffer object includes the first message class and the second message class, as well as the field values ​​of multiple indicator fields in the first message class and the second message class;

[0034] The step of performing secondary aggregation processing on the aggregated data according to a preset data structure to obtain an audit message includes:

[0035] The header of the audit message is determined based on multiple indicator fields and their corresponding values ​​in the first message class.

[0036] The message data portion of the audit message is determined based on multiple indicator fields and their corresponding values ​​in the second message class.

[0037] Secondly, embodiments of this application provide a data reporting toolkit, which includes: a timer interface, a capacity timer, and a distributed message middleware;

[0038] The timer interface is used for external clients to call in order to collect audit data from a preset database;

[0039] The capacity timer is used to pre-aggregate the audit data according to preset rules and preset fields to obtain aggregated data; and based on a lock-free design, to perform secondary aggregation processing on the aggregated data according to a preset data structure to obtain audit messages, which are then cached in a preset cache middleware.

[0040] The distributed message middleware is used to read the audit message from the cache middleware asynchronously; wherein, the distributed message middleware is also used to allow an external data auditing system to asynchronously consume the audit data from it.

[0041] Thirdly, embodiments of this application provide a data auditing platform system, which includes: a data reporting toolkit and a data auditing system;

[0042] The data reporting toolkit is used to implement a data reporting method as provided in the first aspect of this application;

[0043] The data auditing system is used to asynchronously consume audit messages from the data reporting toolkit in order to audit the audit messages.

[0044] The embodiments of this application have the following beneficial effects:

[0045] The method of this application includes: responding to an external client call to a preset timer interface to collect audit data from a preset database using the timer interface; pre-aggregating the audit data according to preset rules and preset fields to obtain aggregated data; performing secondary aggregation processing on the aggregated data according to a preset data structure based on a lock-free design to obtain audit messages, and caching them in a preset caching middleware; asynchronously writing the audit messages in the caching middleware to a preset distributed message middleware; wherein, the distributed message middleware is used for external data auditing systems to asynchronously consume audit data. The lock-free design in this application can improve system performance, and the use of a caching middleware to cache data can decouple the process and implement flow control for reported data. Finally, the audit messages are stored in the distributed message middleware to await asynchronous consumption by the external data auditing system. Therefore, this effectively solves the problems of low performance, high coupling, and poor flexibility in existing data auditing systems. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This paper illustrates a framework diagram of a data auditing platform system according to an embodiment of this application.

[0048] Figure 2 A flowchart of a data reporting method according to an embodiment of this application is shown;

[0049] Figure 3 A second flowchart of the data reporting method according to an embodiment of this application is shown;

[0050] Figure 4 A third flowchart of the data reporting method according to an embodiment of this application is shown.

[0051] Explanation of key component symbols:

[0052] 100 - Data reporting toolkit; 200 - Data auditing system; 300 - Database. Detailed Implementation

[0053] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0054] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0055] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0056] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0057] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0058] To address the issues of existing data auditing / monitoring systems, such as single reporting indicators, low performance, high coupling, and poor flexibility, this application provides a data reporting method, a data reporting toolkit, and a data auditing platform.

[0059] First, this application provides a data auditing platform system, exemplary, such as... Figure 1 As shown, the data auditing platform system includes a data reporting toolkit 100 and a data auditing system 200. The data auditing system 200 connects to the database 300 through the data reporting toolkit 100; the database 300 is used to store audit data.

[0060] The data reporting toolkit 100 (also known as the SDK) is used to read audit data from the database and implement the data reporting method as described in the embodiments of this application.

[0061] The data auditing system consumes audit messages from the data reporting toolkit to audit those messages. For example, audit data includes: log files, specifically the Binlog file, which is used by the database to record all write operations (such as INSERT, UPDATE, DELETE, etc.). The auditing / monitoring system can track data changes by analyzing the Binlog for business auditing or link monitoring.

[0062] The following examples illustrate the data reporting method.

[0063] Figure 2 A flowchart illustrating a data reporting method according to an embodiment of this application is shown. The data reporting method of this application is applicable to reporting audit data to data auditing systems or monitoring systems, etc.

[0064] As an example, this data reporting method includes the following steps:

[0065] The S100 responds to external client calls to a preset timer interface to collect audit data from a preset database using the timer interface.

[0066] This application provides a complete and unified interface for external data auditing systems to call. Exemplarily, the timer interface includes a data acquisition interface and a web application reporting interface; the data acquisition interface is used to collect audit data from a preset database when called. For example, the database path is pre-configured in a configuration file, the data acquisition interface reads the database path from the configuration file, and obtains audit data based on the database path. The web application is used to report click data, application response speed, and record user behavior for each section of the current page. The current page refers to the client's page.

[0067] This application only requires the client to call a unified interface, resulting in low coupling with client code and easy integration of the SDK into the real-time computing pipeline. Clients include, but are not limited to, data auditing systems.

[0068] S200: According to preset rules, the audit data is pre-aggregated based on preset fields to obtain aggregated data.

[0069] To reduce the amount of data transmitted via network I / O, this embodiment of the application pre-aggregates the audit data. Pre-aggregation includes, but is not limited to, merging the audit data according to preset rules based on preset fields contained in the audit data.

[0070] As an example, the audit data is pre-defined with audit fields including data type, data source table, and data source database. The audit data is then merged based on any one of these three criteria.

[0071] In one implementation, step S200 involves pre-aggregating the audit data according to preset rules and preset fields to obtain aggregated data, including:

[0072] S210, based on a preset time interval, determine the unique pre-aggregation key corresponding to each piece of audit data according to preset fields.

[0073] For example, the audit fields in the audit data include: audit data type, audit dataset source table, audit data source library, audit data collection time (Binlog time when audit data is collected), and data end-to-end latency, as shown in Table 1.

[0074] Table 1 Audit Data

[0075]

[0076] The predefined fields in the audit data include the audit data type, the audit dataset source table, and the audit data source library. Any one of these three fields can be used as the pre-aggregation key. For example, the audit data type can be used as the pre-aggregation key.

[0077] S220 assigns a unique corresponding atomic variable object to each pre-aggregation bond.

[0078] Understandably, upon first obtaining a pre-aggregation key, an atomic variable object is generated for that pre-aggregation key, and this atomic variable object is reused for subsequent identical pre-aggregation keys. In other words, before determining a unique corresponding atomic variable object for each pre-aggregation key, it is necessary to check whether a corresponding atomic variable object exists for the pre-aggregation key; if it does not exist, a corresponding atomic variable object is generated; if it exists, the atomic variable object is obtained.

[0079] The method in this embodiment adopts a lock-free design, creating a unique atomic variable object for each pre-aggregation key and assigning the audit data corresponding to that key to the atomic variable. In subsequent processes, audit data with the same pre-aggregation key is pre-aggregated, and the same atomic variable object is reused, thereby reducing unnecessary JVM (Java Virtual Machine) garbage collection.

[0080] In this application, atomic variables can be safely updated by threads. All operations on atomic variables (such as increment, add, etc.) use `compareAndSet` or atomic methods provided by the JDK to ensure concurrency safety, achieving high-concurrency writes without explicit locking. Furthermore, this application can periodically read and reset the values ​​of atomic variables in batches. At fixed time intervals (e.g., once per second), it packages all currently existing keys and their atomic variable values ​​into Protobuf objects. After packaging, the atomic variable values ​​corresponding to these keys are reset to their initial values ​​(e.g., 0). Protobuf, short for Protocol Buffers, is a high-efficiency and flexible data serialization protocol developed by Google. Protocol Buffers is a structured data serialization mechanism used in communication protocols, data storage, and other scenarios.

[0081] Furthermore, to facilitate the retrieval of pre-aggregation keys and their corresponding atomic variable objects, this application employs a dictionary data structure to store the pre-aggregation keys and their corresponding atomic variable objects, resulting in a pre-aggregation dictionary.

[0082] S230: Aggregate all audit data entries with the same pre-aggregation key, and assign the aggregated data to the corresponding atomic variable object. In other words, add the audit data corresponding to the pre-aggregation key to the atomic variable with the same key. Aggregation includes, but is not limited to, merging data.

[0083] For example, the audit data in Table 1 is pre-aggregated, and audit data of the same audit data type are merged. That is, audit data 1 and audit data 3 corresponding to audit data type a in Table 1 are merged, and audit data 4 and audit data 5 corresponding to audit data type c in Table 1 are merged, resulting in the pre-aggregated Map (pre-aggregated dictionary) shown in Table 2.

[0084] Table 2 Pre-aggregated Map

[0085]

[0086] For example, audit data 1 includes: audit data type 1, audit dataset source table 1, audit data source library 1, audit data collection time 1, and data end-to-end latency 1; audit data 3 includes: audit data type 3, audit dataset source table 3, audit data source library 3, audit data collection time 3, and data end-to-end latency 3. Therefore, the atomic variable value corresponding to the pre-aggregation key obtained after pre-aggregation processing is audit data type a, which is: audit data type 1, audit dataset source table 1, audit data source library 1, audit data collection time 1, data end-to-end latency 1, audit data type 3, audit dataset source table 3, audit data source library 3, audit data collection time 3, and data end-to-end latency 3. For example, the value of the atomic variable is represented using Java's atomic reference type (AtomicReference). The data described above is first wrapped to generate a Java object, and then this Java object is managed using Java's atomic reference. This atomic object will be shared by multiple threads during reporting; therefore, using atomic references during aggregation calculations can ensure concurrency safety and performance.

[0087] In this application, the aforementioned metrics can be calculated. For example, using the Binlog time as a starting point, the SDK records the processing time at each subsequent node, and ultimately, the latency from the source to the current node can be calculated. End-to-end data latency = current time - Binlog time.

[0088] The SDK in this application is suitable for high-concurrency environments where multiple threads report audit data simultaneously. Therefore, the cached values ​​in the SDK use atomic variables, and multiple threads share one SDK to ensure thread safety and performance in high-concurrency environments.

[0089] S300, based on a lock-free design, performs secondary aggregation processing on aggregated data according to a preset data structure to obtain audit messages, which are then cached in a preset cache middleware.

[0090] As an example, secondary aggregation processing includes, but is not limited to, packetization and serialization.

[0091] This application achieves process decoupling and reporting flow control by caching audit messages in the caching middleware, ensuring the stability of the SDK and the subsequent scalability of the code.

[0092] Exemplary lock-free design includes mechanisms such as atomic operations and CAS (Compare-And-Swap) instructions to enable concurrent access to shared resources by multiple threads without relying on traditional mutex locks (such as synchronized and ReentrantLock), thereby avoiding performance bottlenecks caused by lock contention.

[0093] In one embodiment, based on a lock-free design, the aggregated data is subjected to secondary aggregation processing according to a preset data structure to obtain an audit message, including:

[0094] S310, based on a preset data structure, periodically batches the corresponding atomic variable values ​​through each atomic variable object and serializes them to obtain the serialized data corresponding to each pre-aggregation key, which serves as the audit message corresponding to each pre-aggregation key.

[0095] S320 initializes the value of each atomic variable after assembling the package. After assembling, it resets the atomic variable values, that is, it resets the specific index value of each atomic variable object to its initial value.

[0096] In one implementation, the caching middleware is a lock-free concurrency framework designed based on the first producer-first consumer (also known as the first queue consumer) model, mainly used for efficient handling of event or message delivery.

[0097] Based on a pre-defined data structure, the corresponding atomic variable values ​​are packaged and serialized in batches at regular intervals using each atomic variable object to obtain the serialized data corresponding to each pre-aggregation key, including:

[0098] S311, according to a preset data structure, periodically batches and combines the values ​​of each atomic variable into the corresponding protocol buffer object. In other words, it converts key-value pairs into attribute values ​​of the protocol buffer object, such as attribute values ​​of a Protobuf object.

[0099] Exemplarily, the preset data structure includes a first message class and a second message class. Both the first and second message classes include multiple indicator fields; the protocol buffer object includes the first and second message classes, as well as the field values ​​of the multiple indicator fields within the first and second message classes. The preset data structure of this application allows for the customization of indicator fields, enabling free configuration of the reported indicator fields.

[0100] Furthermore, the aggregated data is further aggregated according to a preset data structure to obtain an audit message, including:

[0101] The header of the audit message is determined based on multiple indicator fields and their corresponding values ​​in the first message class.

[0102] Based on the multiple indicator fields and their corresponding values ​​in the second message class, determine the message data portion of the audit message.

[0103] S312, based on a preset time interval, obtain the corresponding first producer according to the audit data type of the audit message, and call the serialization method built into the protocol buffer through the first producer to serialize each protocol buffer object respectively through the serialization method.

[0104] As an example, Disruptor is used as the caching middleware. Disruptor is a high-performance queue developed by the UK-based forex trading company LMAX, initially designed to solve the latency problem of in-memory queues (performance tests showed it was on the same order of magnitude as I / O operations). A system based on Disruptor can support 6 million orders per second with a single thread. In the audit SDK of this application, it is mainly used as an internal caching middleware to undertake decoupling / flow control responsibilities.

[0105] Disruptor is a lock-free concurrency framework based on the Disruptor producer-Disruptor consumer model. Protocol buffer objects are not limited to Google Protobuf objects.

[0106] Exemplary, based on a preset data structure, the corresponding atomic variable values ​​are packaged and serialized in batches at regular intervals using each atomic variable object to obtain serialized data corresponding to each pre-aggregation key, including:

[0107] (1) Periodically batch the atomic variable values ​​corresponding to each pre-aggregated bond and combine them into corresponding Google Protobuf objects according to the preset data structure.

[0108] For example, a Protobuf object includes two message classes; the fields in the first message class are used to generate the message header; and the fields in the second message class are used to generate the message data portion.

[0109] (2) Based on the data type of the audit message, the Disruptor producer of the corresponding data type calls the built-in serialization method of Protobuf to serialize each Protobuf object. One Protobuf object is used to generate one audit message. Each Protobuf object includes a pre-structured audit data form; the audit data form includes multiple pre-defined audit indicators.

[0110] For example, the types of pre-defined audit indicators include:

[0111] Specific fields (for generating a single data message header), data latency, data link processing latency, data source type, audit dataset source table, audit data source library, target type of data written, target library of data written, target table of data written, and audit dataset source table and audit data source library.

[0112] The audit message consists of a header and a data portion. The header contains fields for data source type, audit dataset source table, audit data source library, data writing target type, data writing target library, data writing target table, audit dataset source table, and audit data source library. The data portion contains fields for data latency and data link processing latency.

[0113] Understandably, specific fields from the Protobuf object (the header of a single data entry), the data's own latency, and the data link processing latency are used as the header of the audit message. Fields such as data source type, audit dataset source table, audit data source library, target type of data written, target library of data written, target table of data written, audit dataset source table, and audit data source library are used as the data portion of the audit message.

[0114] In one embodiment, the caching middleware further includes a first queue; caching to a preset caching middleware includes:

[0115] The first producer for each audit message type caches the corresponding audit message to the first queue.

[0116] Based on the types of audit messages, multiple types of Disruptor producers are set up. For example, audit message types include delayed audit message types, data volume audit message types, and DDL audit message types. Therefore, Disruptor producers include delayed audit message producers, data volume audit message producers, and DDL audit message producers. The audit message includes information about the audit message type. Exemplarily, the caching middleware is Disruptor, the first queue is the Disruptor queue, and the first queue consumer is the Disruptor consumer. Each audit message type's corresponding Disruptor producer caches the audit messages to the Disruptor queue. Disruptor producers of different audit message types obtain the corresponding type of audit message and cache it to the Disruptor queue. This embodiment of the application uses a high-performance Disruptor queue and LMAX Disruptor to construct a circular buffer, achieving lock-free, thread-safe message passing and supporting high-concurrency writing and consumption.

[0117] In this application, audit data is cached and pre-aggregated, and audit messages are assembled. The assembled audit messages are then sent to the cache middleware and awaited to be asynchronously sent to the distributed message middleware by the consumer.

[0118] S400 uses an asynchronous method to write audit messages from the cache middleware into a preset distributed message middleware;

[0119] Among them, the distributed message middleware is used to allow external data auditing systems to asynchronously consume audit data.

[0120] Furthermore, the caching middleware includes a first queue consumer; the distributed message middleware includes a producer pool, a second queue, and a second consumer.

[0121] The audit messages within the cached middleware are written to the pre-defined distributed message middleware asynchronously, including:

[0122] The first consumer parses the message event tags in the audit message and retrieves the corresponding processing logic from the event handler (also known as the consumer) based on the message event tags. The producer pool is used to provide asynchronous sending capability for each processing logic. The processing logic calls the second producer in the producer pool to realize the asynchronous sending of the audit message to the second queue.

[0123] Exemplary, this application uses a dictionary (such as `HashMap` in Java or `std::map` in C++) to register the mapping relationship between event types and corresponding handlers. Users bind events and handlers to the Map by calling the registration method. The event dispatch logic works by having the system find the corresponding handler from the Map based on the event type when it receives an event, and then executing the processing logic within the handler.

[0124] Exemplarily, after consuming an audit message event from the first queue, the first consumer retrieves the corresponding second producer instance from a pre-built producer pool based on the message event tag; it then uses the asynchronous interface of the second producer instance to retrieve the audit message and store it in the second queue. In one embodiment, the caching middleware is Disruptor, which includes Disruptor consumers; the distributed message middleware is not limited to Kafka, the producer pool is a Kafka producer pool, the second queue is a Kafka queue, and the second consumer is a Kafka consumer.

[0125] The Disruptor consumer retrieves the corresponding processing logic from the preset event handling function Map based on the event tag of the audit message;

[0126] When processing audit data, the corresponding Kafka producer is retrieved from the Kafka producer pool; the Kafka producer is used to send audit messages to the Kafka queue asynchronously.

[0127] like Figure 3 , Figure 4 As shown in the embodiments of this application, specifically, the Disruptor queue is used to receive audit messages; the Disruptor consumer is used to parse the message event tags in the messages and obtain the corresponding processing logic (e.g., the functional logic provided by the asynchronous interface) from the event handler mapping table according to the message event tags; the Kafka producer pool is used to provide asynchronous sending capabilities for each processing logic; the processing logic calls the Kafka producer to asynchronously send audit messages to the specified Topic (Kafka queue). The Kafka producer pool adopts an object pool management strategy, including initialization, reuse, and recycling mechanisms. Event tags include, but are not limited to: data volume audit, DDL operation, permission change, login / logout. It can be understood that the Disruptor consumer includes delayed message consumers, data volume message consumers, and DDL message consumers.

[0128] This application is implemented based on an event-driven architecture and dynamic routing. Specifically, the audit message contains tag information (message event tags), such as "data_volume_audit" and "ddl_operation". The Disruptor consumer dynamically looks up the corresponding processing function in the Map based on the message event tag. This achieves decoupling of event distribution and pluggable extension of logic.

[0129] Furthermore, the method also includes: generating a second producer instance corresponding to each audit message type in the producer pool using an eager initialization method based on at least one preset audit message type. Exemplarily, a Kafka producer instance corresponding to each audit message type is generated in the Kafka producer pool using an eager initialization method based on at least one preset audit message type. The Kafka producer pool uniformly manages multiple Kafka producer instances. The audit message types are pre-configured when using the SDK in this embodiment.

[0130] After consuming audit message events from the Disruptor queue, the Disruptor queue consumer retrieves the corresponding Kafka producer instance from a pre-built Kafka producer pool based on the message event tag. It then uses the asynchronous interface of the Kafka producer instance to obtain the audit message and store it in the Kafka queue. In the real-time computing pipeline, the SDK, as the core component for audit data collection and reporting, must possess high availability, low invasiveness, and strong fault tolerance. The distributed message middleware Kafka is used to achieve data decoupling and asynchronous transmission between systems. However, in practical use, Kafka producers may experience data transmission failures due to network failures, service unavailability, or partition unreachability. Therefore, the SDK needs a mechanism to handle these failures, ensuring that even in the event of partial failure, data is delivered to the target system (Kafka in this embodiment), thereby improving the overall reliability and robustness of the SDK.

[0131] In one embodiment, to build a highly reliable, low-latency audit SDK, if a transmission anomaly is detected during the sending of audit messages to the second queue, an anomaly log is recorded, and the audit message is asynchronously retried until a preset retry limit is reached; wherein, it is guaranteed that the audit message is written semantically at least once. Exemplarily, if a transmission anomaly is detected during the sending of audit messages to the Kafka queue, an anomaly log is recorded, and asynchronous retry is performed until a preset retry limit is reached.

[0132] Asynchronous retry: If an audit message fails to be reported to Kafka, the main thread or current task is not blocked. Instead, the message is retried via an independent thread or event-driven mechanism. This avoids impacting the main process performance due to a single failure and increases the likelihood of data delivery.

[0133] For example, when the SDK calls the send() method of a Kafka producer, if an exception occurs (such as TimeoutException, NetworkException, SerializationException, etc.), it catches these exceptions and logs them. The log content includes: exception type, information about the failed data packet (such as key, topic, timestamp), number of attempts, stack trace, etc., which facilitates subsequent troubleshooting and evaluation of the SDK's robustness and stability.

[0134] Furthermore, the SDK internally configures a maximum number of retries, such as a maximum of 3 retries. An interval (e.g., exponential backoff strategy) is set between each retrieval to reduce instantaneous pressure. If the maximum number of retries is reached and the attempt still fails, the transmission is abandoned, and an alarm mechanism is triggered. Therefore, this embodiment of the application can prevent resource exhaustion caused by infinite retries, while also ensuring that failed requests are recovered as much as possible within a controllable range.

[0135] Furthermore, ensure that the audit message is written to semantics at least once, including:

[0136] This application ensures that audit message data is sent without discarding the original data before successful transmission to Kafka; it retains a copy of the audit message data to be sent; and it clears the cached audit message copy data after successful transmission. Therefore, this application guarantees that audit message data will be sent, although duplication is possible (i.e., the same audit message may be written multiple times). Data loss is prevented (provided that Kafka is configured for persistence and the number of replicas is reasonable).

[0137] This application includes Kafka producer pool management, where all Kafka producers are managed by a unified object pool, avoiding frequent creation and destruction. It retrieves the corresponding Kafka producer based on the audit message type, improving performance and preventing resource leaks. This application also includes an asynchronous sending mechanism, where processing logic calls Kafka producers for asynchronous sending, improving overall response speed. It supports callback or failure retry mechanisms to ensure data reliability.

[0138] The audit SDK in this application features ease of use, asynchronous operation, low coupling (minimizing intrusion into the caller's code), and high flexibility (customizable metrics). This application provides a single API, meaning that callers only need to use a unified API and perform simple configurations (e.g., configuring audit messages to correspond to Kafka producer logic; configuring the database metadata the SDK needs to collect). The SDK service provider will then match the configuration with the corresponding implementation, reporting data asynchronously. This minimizes the impact on ETL performance and improves overall performance. ETL performance refers to the efficiency and stability of ETL (Extract, Transform, Load) tools during data processing. Key performance indicators for ETL tools include processing speed, resource consumption, error rate, and stability.

[0139] This application provides a data reporting toolkit, which, by way of example, includes: a timer interface, a capacity timer, and a distributed message middleware.

[0140] The timer interface is used by external clients to collect audit data from a preset database.

[0141] The capacity timer is used to pre-aggregate audit data according to preset rules and preset fields to obtain aggregated data; based on a lock-free design, it performs secondary aggregation processing on the aggregated data according to a preset data structure to obtain audit messages, which are then cached in a preset cache middleware.

[0142] Distributed message middleware is used to read audit messages asynchronously from cache middleware; it also serves as a means for external data auditing systems to asynchronously consume audit data.

[0143] In some embodiments, the capacity timer includes a pre-aggregator.

[0144] Prepolymerizer, used for:

[0145] Based on preset time intervals, a unique pre-aggregation key is determined for each piece of audit data according to preset fields;

[0146] For each pre-aggregation bond, assign a unique corresponding atomic variable object;

[0147] Aggregate all audit data entries with the same pre-aggregation key, and assign the resulting aggregated data to the corresponding atomic variable object.

[0148] Furthermore, the pre-aggregator is specifically used to: check if a pre-aggregation key has a corresponding atomic variable object; if it does not exist, generate a corresponding atomic variable object; if it exists, obtain the atomic variable object.

[0149] Furthermore, the secondary aggregation process includes packet assembly and serialization; the capacity timer includes a message combiner.

[0150] Message combiner, used for:

[0151] Based on a preset data structure, the corresponding atomic variable values ​​are packaged and serialized in batches at regular intervals through each atomic variable object to obtain the serialized data corresponding to each pre-aggregation key, which serves as the audit message corresponding to each pre-aggregation key.

[0152] After assembling the package, initialize the value of each atomic variable.

[0153] Furthermore, the message combiner is specifically used for:

[0154] According to the preset data structure, the values ​​of each atomic variable are periodically and batch-combined into the corresponding protocol buffer objects;

[0155] Based on a preset time interval, the corresponding first producer is obtained according to the audit data type of the audit message. The serialization method built into the protocol buffer is called through the first producer, and each protocol buffer object is serialized through the serialization method.

[0156] Furthermore, the capacity timer includes a first producer, and the cache middleware also includes a first queue.

[0157] The first producer, used for:

[0158] The corresponding audit message type is cached in the first queue according to each audit message type.

[0159] Furthermore, the data reporting toolkit also includes a distributed message middleware, and the caching middleware includes a first consumer; the distributed message middleware includes a producer pool, a second queue, and a second consumer. After consuming audit message events in the first queue, the first consumer retrieves the corresponding second producer instance from the pre-built producer pool based on the message event tag; it then uses the asynchronous interface of the second producer instance to retrieve the audit message and store it in the second queue.

[0160] The data reporting toolkit also includes an initialization module. This module is used to generate a second producer instance corresponding to each audit message type in the producer pool, based on at least one preset audit message type, using an eager initialization approach.

[0161] The data reporting toolkit also includes a retry module. This module is used to record an exception log and asynchronously retry sending the audit message if an exception is detected during the process of sending the audit message to the second queue, until the preset retry limit is reached. It ensures that the audit message is written to the semantics at least once.

[0162] Furthermore, the preset data structure includes a first message class and a second message class; both the first message class and the second message class include multiple indicator fields; the protocol buffer object includes the first message class and the second message class, as well as the field values ​​of multiple indicator fields in the first message class and the second message class;

[0163] Capacity timers are also used for:

[0164] The header of the audit message is determined based on multiple indicator fields and their corresponding values ​​in the first message class.

[0165] Based on the multiple indicator fields and their corresponding values ​​in the second message class, determine the message data portion of the audit message.

[0166] This application has the following advantages:

[0167] (1) Key-based atomic variable caching mechanism; supports key aggregation of any dimension; dynamically expands the key set and allocates atomic variables as needed; avoids global locks and improves concurrency performance.

[0168] (2) Lock-free micro-aggregation + timed packaging + asynchronous flushing pipeline; there are no synchronization blocks or wait / notify logic in the overall process; packaging and reporting logic are decoupled through Disruptor; end-to-end asynchronous, non-blocking, high-performance audit collection is achieved.

[0169] (3) Lightweight access + automatic reset mechanism; the caller only needs to call the unified API to update the atomic variable corresponding to the key; the capacity timer automatically manages packaging and reset without interfering with business logic.

[0170] It is understood that the apparatus in this embodiment corresponds to the data reporting method in the above embodiments, and the options in the above embodiments are also applicable to this embodiment, so they will not be described again here.

[0171] This application also provides a terminal device, exemplary of which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the functions of the data reporting method described above or the various modules in the data reporting toolkit described above.

[0172] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0173] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store computer programs, and the processor can execute the computer programs accordingly after receiving execution instructions.

[0174] This application also provides a computer-readable storage medium for storing the computer program used in the aforementioned terminal device. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, in alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0176] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0177] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0178] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data reporting method, characterized in that, The method includes: In response to an external client calling a preset timer interface, audit data is collected from a preset database using the timer interface; According to preset rules, the audit data is pre-aggregated based on preset fields to obtain aggregated data. Specifically, this includes: determining a unique pre-aggregation key for each audit data entry based on preset time intervals and preset fields; assigning a unique atomic variable object to each pre-aggregation key; aggregating all audit data entries with the same pre-aggregation key and assigning the resulting aggregated data to the corresponding atomic variable object; wherein, assigning a unique atomic variable object to each pre-aggregation key includes: checking if a corresponding atomic variable object exists for the pre-aggregation key; if not, generating a corresponding atomic variable object; if it exists, obtaining the atomic variable object. Based on a lock-free design, the aggregated data is subjected to secondary aggregation processing according to a preset data structure to obtain audit messages, which are then cached in a preset cache middleware. Specifically, this includes: based on the preset data structure, the corresponding atomic variable values ​​are packaged and serialized in batches at regular intervals through each atomic variable object to obtain serialized data corresponding to each pre-aggregation key, which serves as the audit message corresponding to each pre-aggregation key. The audit message in the cache middleware is written to a preset distributed message middleware in an asynchronous manner; the distributed message middleware is used for external data auditing systems to asynchronously consume the audit data.

2. The data reporting method according to claim 1, characterized in that, Based on the preset data structure, the process of periodically batching and serializing the corresponding atomic variable values ​​through each atomic variable object to obtain serialized data corresponding to each pre-aggregation key includes: According to the preset data structure, each atomic variable value is periodically and batch-combined into a corresponding protocol buffer object; Based on a preset time interval, the corresponding first producer is obtained according to the audit data type of the audit message. The serialization method built into the protocol buffer is called through the first producer, and each protocol buffer object is serialized through the serialization method.

3. The data reporting method according to claim 2, characterized in that, The caching middleware also includes a first queue; The caching to a preset caching middleware includes: The first producer for each audit message type caches the corresponding audit message to the first queue.

4. The data reporting method according to claim 3, characterized in that, The caching middleware includes a first consumer; the distributed message middleware includes a producer pool, a second queue, and a second consumer. The step of asynchronously writing the audit message from the cache middleware into a preset distributed message middleware includes: The first consumer parses the message event tag in the audit message and obtains the corresponding processing logic from the event handler based on the message event tag; wherein, the producer pool is used to provide asynchronous sending capability for each processing logic; the processing logic calls the second producer in the producer pool to realize asynchronous sending of the audit message to the second queue; Specifically, after consuming the audit message event in the first queue, the first consumer obtains the corresponding second producer instance from the pre-built producer pool according to the message event label; and uses the asynchronous interface of the second producer instance to obtain the audit message and store it in the second queue.

5. The data reporting method according to claim 4, characterized in that, The method also includes one or more of the following three items: First item: Based on at least one preset audit message type, generate a second producer instance corresponding to each audit message type in the producer pool using the eager initialization method; The second item: If an error is detected during the process of sending the audit message to the second queue, an error log is recorded, and the audit message is retried asynchronously until the preset retry limit is reached; wherein, the audit message is guaranteed to be written with semantics at least once; The third item: The preset data structure includes a first message class and a second message class; both the first message class and the second message class include multiple indicator fields; the protocol buffer object includes the first message class and the second message class, as well as the field values ​​of multiple indicator fields in the first message class and the second message class; The step of performing secondary aggregation processing on the aggregated data according to a preset data structure to obtain an audit message includes: The header of the audit message is determined based on multiple indicator fields and their corresponding values ​​in the first message class. The message data portion of the audit message is determined based on multiple indicator fields and their corresponding values ​​in the second message class.

6. A data reporting toolkit, characterized in that, The data reporting toolkit includes: a timer interface, a capacity timer, and a distributed message middleware; The timer interface is used for external clients to call in order to collect audit data from a preset database; The capacity timer is used to pre-aggregate the audit data according to preset rules and preset fields to obtain aggregated data; based on a lock-free design, it performs secondary aggregation processing on the aggregated data according to a preset data structure to obtain audit messages, and caches them in a preset cache middleware; specifically, the capacity timer is used to determine a unique pre-aggregation key for each piece of audit data according to preset fields based on preset time intervals; determine a unique atomic variable object for each pre-aggregation key; aggregate all audit data with the same pre-aggregation key, and assign the obtained aggregated data to the corresponding atomic variable object; wherein, determining a unique atomic variable object for each pre-aggregation key includes: checking whether a corresponding atomic variable object exists for the pre-aggregation key; if it does not exist, generating a corresponding atomic variable object; if it exists, obtaining the atomic variable object; The capacity timer is further used to, based on the preset data structure, periodically batch-package and serialize the corresponding atomic variable values ​​through each atomic variable object to obtain the serialized data corresponding to each pre-aggregation key, which serves as the audit message corresponding to each pre-aggregation key; The distributed message middleware is used to read the audit message from the cache middleware asynchronously; wherein, the distributed message middleware is also used to allow an external data auditing system to asynchronously consume the audit data from it.

7. A data auditing platform system, characterized in that, The data auditing platform system includes: a data reporting toolkit and a data auditing system; The data reporting toolkit is used to implement the data reporting method as described in any one of claims 1-5; The data auditing system is used to asynchronously consume audit messages from the data reporting toolkit in order to audit the audit messages.

Images