Distributed database auditing method and device, storage medium and program product
By introducing an audit twin library into the distributed database, the decoupling of auditing and business logic is achieved, solving the problems of audit integrity and performance efficiency in existing technologies, and improving the system's throughput and response speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CETC JINCANG (BEIJING) TECH CO LTD
- Filing Date
- 2026-02-10
- Publication Date
- 2026-05-12
AI Technical Summary
Existing distributed database auditing solutions struggle to balance audit integrity with system performance efficiency, especially in high-concurrency scenarios where excessive computational resources are consumed, cross-shard operation records are missed, and resources are wasted.
An audit twin is introduced to keep metadata synchronized with the distributed database, but does not store user data. It independently parses and generates audit records, and ensures real-time performance and accuracy through a metadata synchronization mechanism, thus decoupling audit from business logic.
It achieves full audit coverage, low performance impact, and efficient resource utilization, generates complete audit logs, improves system throughput and response speed, and reduces computing and storage resource consumption.
Smart Images
Figure CN122019671A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed systems, and more particularly to a distributed database auditing method, device, storage medium, and program product. Background Technology
[0002] In distributed databases, compute nodes handle user requests, while storage nodes distribute data through data sharding. With increasingly stringent data security compliance requirements, comprehensive database auditing has become a critical need.
[0003] Currently, auditing functionality in distributed databases is primarily implemented by deploying auditing modules on compute or storage nodes. At the compute node level, the auditing module typically records information such as SQL operations, accessed objects, and execution times that pass through that node. At the storage node level, auditing captures logs of operations involving data shards on that node based on the actual data storage location.
[0004] However, the technical solutions described above are difficult to balance audit integrity and system performance efficiency. Summary of the Invention
[0005] This application provides a distributed database auditing method, device, storage medium, and program product to solve the technical problems of audit integrity and system performance efficiency.
[0006] In a first aspect, embodiments of this application provide a distributed database auditing method applied to auditing a twin database. The auditing twin database maintains metadata synchronization with the distributed database but does not store user data from the distributed database. The metadata includes the table structure, index definitions, and sharding rules of the distributed database. The method includes:
[0007] Receive SQL requests from distributed database computing nodes. The SQL requests are used to execute either a first operation or a second operation. The first operation is to modify user data, and the second operation is to not modify user data.
[0008] Perform the following operations based on the metadata synchronized with the distributed database:
[0009] If the SQL request is the first operation, parse the semantic scope of the first operation without actually performing data changes, and generate an audit record based on the parsing result;
[0010] If the SQL request is a second operation, process the second operation to obtain operation context information, and generate an audit log based on the operation context information.
[0011] In this embodiment, by introducing an audit twin library component, the technical effects of full audit coverage, low performance impact, and efficient resource utilization are achieved. Specifically, regardless of whether the SQL request involves cross-shard operations, a corresponding audit record can be generated in the audit twin library, thereby solving the problem that existing storage node audits may miss cross-shard operations. Simultaneously, audit processing is completed in an independent audit twin library, without consuming computing resources on the compute nodes or affecting the data operation performance of the storage nodes. Especially for data modification operations, the audit twin library only parses the semantics without actually executing them, resulting in faster processing speeds. Furthermore, compared to the resource waste caused by deploying audit functionality on all storage nodes in a distributed database, this method saves computing and storage resources by centrally processing all audit requests through a single audit twin library.
[0012] In one possible implementation, the audit twin maintains metadata synchronization with the distributed database in the following way:
[0013] Meta-information is retrieved from the distributed database during the initialization of the audit twin database;
[0014] When the metadata of the distributed database changes, the metadata of the audit twin database is updated synchronously.
[0015] In this implementation, the metadata synchronization mechanism enables the audit twin database to obtain a structure definition consistent with the production database in real time and accurately, providing a reliable foundation for accurate SQL semantic parsing. At the same time, the monitoring-response and incremental update strategy adopted by this mechanism reduces performance interference and resource consumption on the production system while ensuring real-time synchronization.
[0016] In one possible implementation, the semantic scope of parsing the first operation includes:
[0017] Identify the data tables and fields involved in the first operation;
[0018] Analyze the conditional constraints of the first operation;
[0019] Logical estimation of the scope of influence based on conditional constraints.
[0020] In this implementation, through layered and structured analysis steps, the audit twin library can perform precise semantic understanding and logical estimation of the intent and potential impact of data modification operations without accessing actual user data. This not only identifies the specific targets (such as tables and fields) and precise conditions of the operation, but also infers the logical scope of the affected data, thereby generating audit logs that contain business semantics and meet compliance requirements. This solves the problems of insufficient information and inability to reflect the business intent of operations in traditional audit logs.
[0021] In one possible implementation, the second operation includes at least one of Data Query Language (DQL) operation, Data Definition Language (DDL) operation, and Data Control Language (DCL) operation.
[0022] In this implementation, by clearly defining the scope of the second operation, the audit twin library can perform comprehensive and categorized fine-grained auditing of all read-only or administrative non-data change operations in the database system.
[0023] In one possible implementation, the method further includes:
[0024] During the parsing of the first operation or the processing of the second operation, it is determined whether the SQL request is a cross-shard operation based on the sharding rules.
[0025] In response to an SQL request that is a cross-shard operation, the cross-shard operation information is identified in the generated audit log.
[0026] In this implementation, by identifying and marking cross-shard operations, key dimensions of information are provided for operation tracing, performance analysis, and compliance auditing in a distributed environment, solving the problem of insufficient perception of data flow topology in traditional auditing in distributed scenarios.
[0027] In one possible implementation, the audit log includes at least one of the following information: operation type, operation time, operation user, and data objects involved.
[0028] This approach enhances the standardization and completeness of audit outputs by clearly defining the core information fields that audit logs must include, providing directly usable, structured key evidence for operational traceability, security incident analysis, and compliance verification.
[0029] In one possible implementation, the method further includes:
[0030] Maintain the cache of processing results;
[0031] For the received SQL request, check if there is a cached processing result in the cache;
[0032] If it exists, generate an audit log based on the cached processing results;
[0033] If it does not exist, perform the corresponding processing operation to obtain a new processing result, generate an audit record, and store the new processing result in the cache;
[0034] The processing results include parsing results or operation context information.
[0035] In this implementation, by implementing the aforementioned caching strategy, in real-world application scenarios with a large number of repetitive or patterned SQL requests, it can effectively avoid redundant parsing and computation of requests with the same semantics. By directly reusing the intermediate semantic analysis results already cached locally, the audit twin library can reduce CPU resource consumption and redundant memory usage, thereby improving the throughput and response speed of the audit system under high concurrency loads.
[0036] In a second aspect, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;
[0037] The memory stores instructions that the computer executes;
[0038] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.
[0039] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.
[0040] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects. Attached Figure Description
[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0042] Figure 1 A flowchart illustrating a distributed database auditing method provided in an embodiment of this application;
[0043] Figure 2 This is a schematic diagram of the architecture of a distributed database auditing system provided in an embodiment of this application.
[0044] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0045] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0046] It should be noted that the distributed database auditing method, device, storage medium, and program product provided in this application can be used in the distributed field, or in any field other than distributed. This application does not limit the application field of the distributed database auditing method, device, storage medium, and program product.
[0047] This application can be applied to scenarios that employ a distributed database architecture and have strict auditing requirements for data operations. Specifically, it can be applied to database systems that need to handle high-concurrency queries and transaction requests at the computing layer and achieve distributed data storage and horizontal scaling through data sharding technology at the storage layer. Such systems are commonly found in critical information infrastructures such as enterprise-level financial transaction centers, e-government data sharing platforms, and high-throughput IoT data processing centers. Their business characteristics include large data scale, complex access patterns, and high requirements for service continuity and response performance.
[0048] In these scenarios, to meet the data security supervision and operation tracking requirements stipulated by relevant industry standards, the system must have the ability to record all user operations on the database in real time and accurately. The recorded audit information typically includes key metadata such as operation type (e.g., query, insert, modify, delete), corresponding SQL statement, accessed data objects (e.g., tables, rows, columns), execution timestamp, executing user, and application identifier. This high standard of database auditing capability is precisely the typical application environment that this application's technical solution aims to adapt to.
[0049] For the above application scenarios, existing technical solutions typically implement the auditing function of distributed databases in the following ways:
[0050] I. Auditing Implementation Based on Computing Nodes
[0051] In this implementation, the auditing function is built into the compute nodes of the distributed database. Specifically, when a user's SQL request arrives at the compute node, the compute node synchronously triggers the audit logging mechanism during the SQL parsing, query optimization, and execution plan generation processes. This mechanism captures information such as the SQL statement text, the identifier of the accessed data object, the operation type, the execution timestamp, and the user identifier, and generates corresponding audit logs. This approach eliminates the need for a separate dedicated auditing component, and the auditing behavior is highly synchronized with the query execution process.
[0052] II. Auditing Implementation Based on Storage Nodes
[0053] In this implementation, the auditing function is deployed on the storage nodes of the distributed database. Since the data is distributed across multiple storage nodes in a sharded manner, the auditing module needs to run locally on each storage node. When a data access operation reaches a storage node, the auditing module on that node will log the operation involving the data shards of that node. To achieve complete operation coverage, auditing instances are typically deployed on all storage nodes, forming a multi-node distributed audit log generation system.
[0054] However, the solution described above has the following technical problems:
[0055] First, in compute node-based auditing implementations, the generation of audit logs is tightly coupled with the SQL execution flow. This requires compute nodes to simultaneously complete audit recording while processing business logic, thus consuming computing and memory resources that should be used for query processing. Especially in high-concurrency scenarios, the real-time execution of the auditing function impacts the overall database performance, becoming a constraint on system throughput and response latency.
[0056] Secondly, in the implementation of auditing based on storage nodes, due to the distributed nature of data sharding, a single storage node can only record operations involving its local shard. If auditing is only deployed on some nodes, it cannot cover data operations involving multiple nodes, such as cross-shard queries, resulting in missing audit records. If auditing is deployed on all storage nodes, although full coverage of operations can be achieved, it will lead to redundant hardware resource configuration, increased operational complexity, and scattered audit logs that are difficult to manage and analyze effectively.
[0057] Furthermore, existing solutions deeply embed audit logic into the database kernel architecture. This tightly coupled design means that any adjustment to the audit function may affect the stability of the core database business and also limits the independent evolution capability of the audit system.
[0058] The distributed database auditing method provided in this application aims to solve the aforementioned technical problems of existing technologies. By constructing a dedicated audit twin library in the distributed database architecture that is synchronized with the distributed database metadata but does not store actual user data, the physical and logical decoupling of auditing functions from the core business logic of the distributed database is achieved.
[0059] Specifically, upon receiving a user's SQL request, the compute node executes two independent processing paths in parallel: one path hands the SQL request over to the distributed database for normal data querying and transaction processing; the other path forwards the SQL request to the audit twin. Upon receiving the SQL request, the audit twin performs simulated parsing and virtual execution within its own metadata environment, consistent with the distributed database, and generates an audit log containing complete information such as the SQL statement, the operation object, and timestamps. Because the audit twin only carries the metadata structure and audit logic, and does not actually perform data modification operations or persist user business data, its consumption of computing and storage resources is extremely low.
[0060] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.
[0061] First, combine Figure 1 This application introduces a distributed database auditing method provided in its embodiments. Figure 1 This is a flowchart illustrating a distributed database auditing method provided in an embodiment of this application.
[0062] It should be noted that the execution entity in this embodiment is an audit twin, which is a dedicated audit component that maintains metadata synchronization with the distributed database but does not store user data. The metadata includes the table structure, index definitions, and sharding rules of the distributed database. Furthermore, the audit twin is independent of the distributed database's compute and storage nodes and is specifically responsible for generating audit records.
[0063] like Figure 1 As shown, the method includes the following steps:
[0064] S101, Receive SQL requests from distributed database computing nodes.
[0065] Specifically, after receiving an SQL request from a user or application, the computing node of the distributed database forwards the SQL request to two destinations simultaneously: one is to forward it to the corresponding storage node to perform actual data operations; the other is to forward it to the audit twin database for audit processing.
[0066] This SQL request is used to perform either a first operation or a second operation. The first operation is an operation that modifies user data, such as a Data Manipulation Language (DML) operation. The second operation is an operation that does not modify user data, such as a Data Query Language (DQL), Data Definition Language (DDL), or Data Control Language (DCL) operation.
[0067] S102. If the SQL request is the first operation, parse the semantic scope of the first operation without actually performing data changes, and generate an audit record based on the parsing result.
[0068] When the SQL request is an operation to modify user data (the first operation), the audit twin database performs the following steps:
[0069] First, SQL semantics are parsed based on synchronized metadata. This involves using metadata such as table structure and index definitions synchronized from the distributed database to understand the complete semantics of the SQL request. For example, for an UPDATE statement, this determines the table being updated, the fields, the update conditions, and the updated values.
[0070] Next is parsing the semantic impact of the operation, which involves analyzing which data the operation will affect, estimating the amount of data affected, and the shards involved, but without actually executing the data changes. For example, for "UPDATE users SET status=1WHERE age>30", the parsing will update the status field of users in the users table whose age is greater than 30.
[0071] Finally, an audit record is generated based on the parsing results. This involves combining the semantic information, scope of impact, operation time, and operation user information obtained from the parsing to generate a complete audit record, which is then stored in a separate audit storage.
[0072] S103. If the SQL request is a second operation, process the second operation to obtain operation context information, and generate an audit record based on the operation context information.
[0073] When the SQL request is an operation that does not modify user data (the second operation), the audit twin database performs the following steps:
[0074] The first step is to process the corresponding operations to obtain the operation context information. Specifically, for query operations (DQL), the query semantics, involved fields, and query conditions are parsed; for structure change operations (DDL), the object information for creation or deletion is parsed; and for permission operations (DCL), the authorized object and permission information are parsed.
[0075] Then, audit logs are generated based on the operation context information, that is, the obtained operation context information is combined with operation metadata (time, user, etc.) to generate audit logs. For DDL and DCL operations, the audit twin can perform corresponding metadata changes in its own environment to verify the legality of the operation.
[0076] The distributed database auditing method provided in this embodiment achieves full audit coverage, low performance impact, and efficient resource utilization by introducing an audit twin library component. Specifically, regardless of whether the SQL request involves cross-shard operations, a corresponding audit record can be generated in the audit twin library, thus solving the problem that existing storage node audits may miss cross-shard operations. Simultaneously, audit processing is completed in an independent audit twin library, without consuming computing resources on compute nodes or affecting the data operation performance of storage nodes. Especially for data modification operations, the audit twin library only parses the semantics without actually executing them, resulting in faster processing speeds. Furthermore, compared to the resource waste caused by deploying auditing functions on all storage nodes in a distributed database, this method saves computing and storage resources by centrally processing all audit requests through a single audit twin library.
[0077] The steps in the above embodiments can be performed at any time. Figure 2 The system architecture shown is implemented in detail. Specifically, steps S101-S103 are all executed by the audit twin database, the compute node is responsible for forwarding SQL requests, and the storage node is responsible for the actual data operations.
[0078] To more clearly illustrate the technical solution of this application, the following is combined with... Figure 2 The system architecture of this application is described in detail. Figure 2 This is a schematic diagram of the architecture of a distributed database auditing system provided in an embodiment of this application.
[0079] like Figure 2 As shown, the auditing system consists of two parts: a distributed database and an audit twin database. The distributed database adopts a classic master-slave architecture, consisting of one compute node and multiple (n as shown in the figure) storage nodes. The compute node acts as the entry point, responsible for receiving and parsing user SQL requests and distributing specific data operation instructions to the corresponding storage nodes for execution.
[0080] The audit twin is a dedicated database instance that is independent of the distributed database. Its core feature is that it maintains completely consistent metadata (such as table structure and index definition) with the distributed database, but it does not store any actual user business data.
[0081] The independence of the entire audit process is Figure 2 The key to the architecture shown is that it completely isolates audit operations from core business operations at the resource and execution levels. Specifically, when a user's SQL request arrives at the compute node, the compute node forwards the request in two directions: on the one hand, it is normally distributed to the storage node to execute business operations, ensuring that the business process is not disrupted; on the other hand, it is synchronously forwarded to the audit twin. After receiving the SQL request, the audit twin does not actually execute data changes, but uses synchronized metadata to "simulate execution" or parse the SQL.
[0082] In this way, the audit twin database can accurately analyze the complete contextual information of SQL operations, such as semantics and the range of data affected, and then generate audit records containing information such as the operator, operation time, and operation object.
[0083] The purpose of this design is to achieve complete decoupling between auditing and business operations. Since all auditing logic is completed in an independent audit twin, it does not impose additional performance load on the distributed database's compute and storage nodes, thus avoiding interference with business operations. Furthermore, due to the real-time consistency of metadata, the audit twin does not need to piece together and aggregate information from multiple scattered storage nodes to directly generate accurate and complete audit records, improving auditing efficiency and reducing system complexity.
[0084] Optionally, to improve the accuracy of auditing, embodiments of this application provide a mechanism for synchronizing metadata between the audit twin and the distributed database. This mechanism enables the audit twin to have a database structure view consistent with the production environment, thereby correctly parsing SQL requests.
[0085] During the startup or initialization phase of the audit twin database, a dedicated connection is first established with the management node of the distributed database, and a complete metadata snapshot, including all table structures, index definitions, and sharding rules, is obtained in batches. This snapshot is then persistently stored locally to build a metadata database with a completely identical structure. This process only replicates the database structure definition and does not involve any user business data.
[0086] During the continuous operation of the audit twin, the system monitors changes to the metadata of the distributed database in real time through event-driven or polling mechanisms. When a DDL operation occurs, the audit twin receives a change notification immediately, obtains the specific change details, and applies the same structural changes to its local metadata database, thus maintaining real-time synchronization between the two. For example, if a new field is added to the production database, the audit twin can quickly update its local table definition, ensuring that all subsequent SQL requests involving that field can be correctly parsed and generate complete audit records.
[0087] The synchronization mechanism also has fault tolerance and self-recovery capabilities. In case of abnormal situations such as network interruption or synchronization conflict, the system has the ability to automatically retry, compare versions and repair, and prioritize the metadata of the distributed database to ensure eventual consistency.
[0088] Through the metadata synchronization mechanism, the audit twin database can obtain a structure definition consistent with the production database in real time and accurately, providing a reliable foundation for accurate SQL semantic parsing. At the same time, the monitoring-response and incremental update strategy adopted by this mechanism reduces performance interference and resource consumption on the production system while ensuring real-time synchronization.
[0089] Optionally, for the first operation (i.e., the operation of modifying user data), this application embodiment also provides a specific implementation method for semantic impact scope resolution. When the audit twin determines that the SQL request is the first operation, its resolution process follows the following three levels.
[0090] First, identify the data objects involved in the operation. The audit twin performs lexical and syntactic analysis on the SQL statement based on its own maintained metadata synchronized with the distributed database, identifying the target data table directly affected by the operation and the affected fields within that table. For example, for an UPDATE statement, this step can clearly identify the name of the table being updated and the fields to be modified listed in the SET clause.
[0091] Secondly, the audit twin analyzes the conditions and constraints of the operation. By analyzing the WHERE clauses, JOIN conditions, or other filtering logic in the SQL statement, the audit twin transforms these conditions into understandable structured constraint information. For example, for the condition "WHERE department='Sales' AND status='active'", this step will resolve two equality constraints, clearly indicating that the operation only applies to records in the "Sales" department with a status of "active".
[0092] Finally, a logical estimation of the scope of impact is performed. Based on the data objects and constraints obtained in the first two steps, the audit twin database, combined with the metadata of the table structure (such as field types and indexes), performs logical reasoning and estimation of the range of data records that may be affected by this operation.
[0093] It should be noted that since the audit twin does not store actual user data, this estimation is not based on precise data statistics, but rather on a logical judgment based on semantics and constraints. For example, based on the parsed conditions, it is estimated that the operation will affect all active employee records in the 'Sales' department. This estimation result is used as key semantic information for understanding the scope of the operation's impact and recorded in the audit log. Thus, without accessing the actual data, an audit record with business semantics that meets compliance review requirements is generated.
[0094] Through layered and structured analysis steps, the audit twin library can accurately understand the intent and potential impact of data modification operations without accessing actual user data. This not only identifies the specific targets (such as tables and fields) and precise conditions of the operation, but also infers the logical scope of the affected data, thereby generating audit logs that contain business semantics and meet compliance requirements. This solves the problems of insufficient information and inability to reflect the business intent of operations in traditional audit logs.
[0095] Optionally, this embodiment further clarifies the specific operation types covered by the second operation (i.e., the operation that does not modify user data), including at least one of DQL operation, DDL operation and DCL operation.
[0096] For DQL operations, this mainly refers to SELECT statements. When the audit twin receives such operations, the focus is on parsing the semantic intent of the query, such as identifying the target table, fields, filtering conditions (WHERE clause), sorting and grouping methods (ORDER BY, GROUP BY), and join relationships, in order to understand what data the user wants to view and generate audit logs based on this.
[0097] For DDL operations, including statements that create, modify, or delete database structure objects, such as CREATE TABLE, ALTER TABLE, DROP INDEX, etc., the audit twin database, when handling such operations, not only records the operation itself, but may also utilize metadata synchronized with the master database to simulate the execution of the DDL statement in its own environment to verify syntax correctness and accurately record the complete context of database structure changes.
[0098] For DCL operations, which mainly refer to permission management statements such as GRANT (grant) and REVOKE (revoke) permissions, the audit twin database parses the users, roles, permission objects, and specific permission types involved in such operations, generating detailed records. This is crucial for meeting security compliance audit requirements.
[0099] By clearly defining the scope of the second operation, the audit twin library can perform comprehensive and categorized fine-grained auditing of all read-only or administrative non-data change operations in the database system.
[0100] Optionally, this embodiment provides an enhanced method for identifying and auditing cross-shard operations. During the semantic analysis process of parsing the first operation or processing the second operation in the audit twin, a cross-shard judgment process is initiated simultaneously. This judgment is based on the sharding rules maintained by the audit twin, which are strictly consistent with the distributed database.
[0101] Specifically, the audit twin first extracts key data identifiers or condition ranges involved in the operation from the parsed SQL semantics, and then substitutes these identifiers or ranges into the sharding rules for calculation. If the calculation result indicates that the operation needs to access or affect multiple different storage shards, the operation is determined to be a cross-shard operation. For example, for a query containing the condition WHERE user_id IN (1001, 1002, 1003), the audit twin calculates the shard to which each user_id belongs according to the sharding rules. If it finds that these IDs are distributed in two or more different shards, the operation is determined to be a cross-shard operation.
[0102] When an SQL request is determined to be a cross-shard operation, the audit twin will add a corresponding cross-shard operation identifier and related information to the audit record generated for that request. This identifier can be a Boolean flag or contain more detailed information, such as the specific number of shards involved or a list of shard identifiers. This enhanced information is integrated into fixed fields or extended metadata in the audit record, so that the final audit record not only reflects what the operation did, but also clearly characterizes how the operation was executed in a distributed architecture.
[0103] By identifying and tagging cross-shard operations, key dimensions of information are provided for operation tracing, performance analysis, and compliance auditing in distributed environments, solving the problem of insufficient perception of data flow topology in traditional auditing scenarios.
[0104] Optionally, embodiments of this application further define the basic information fields that the audit log should include. These fields constitute the core metadata that meets basic audit and compliance requirements. Specifically, they include:
[0105] Operation type: Records the specific operation category of the SQL request, such as SELECT, INSERT, UPDATE, DELETE, or CREATE TABLE, GRANT, etc. This information directly reflects the intent of the user or application. For example, it can clearly distinguish between a data query and a structural change.
[0106] Operation Time: Records the timestamps of when an SQL request is received and processed. Timestamps accurate to milliseconds or microseconds are the foundation for establishing strict operation sequences, performing causal analysis, and meeting the legal requirements for the immutability of operation records.
[0107] User Operation: Records the identity of the database user who executed the SQL request, providing direct evidence for liability determination and behavior correlation. For example, in security incident investigations, it can quickly identify the specific person responsible.
[0108] Data objects involved: Records the specific database objects targeted by this operation, such as table names, view names, and more granular field names. For cross-shard operations, this information can be combined with sharding rules to further characterize the distribution of data objects.
[0109] These fields work together to ensure that each generated audit record can structurally carry complete operational semantics. For example, for a request to update employee salaries, the final record generated by the audit twin will explicitly include: operation type "UPDATE", operation user "hr_admin", operation time "2023-10-27 14:30:15.456", and the data object involved "salary field of the employees table". When the operation is a cross-shard operation, the record will also include the cross-shard identifier defined in the above embodiments. This standardized information structure enables audit records to clearly represent the core audit questions of "who, when, what data, and what operation was performed".
[0110] By clearly defining the core information fields that audit records must include, the standardization and completeness of audit outputs are improved, providing directly usable, structured key evidence for operation traceability, security incident analysis, and compliance verification.
[0111] Optionally, this embodiment provides a caching strategy for improving audit processing performance. This strategy is implemented in an audit twin database, and its core lies in establishing and maintaining a cache that stores the mapping relationship between SQL requests and their corresponding processing results. The technical meaning of these processing results may include the following:
[0112] For operations that modify user data (the first operation), the processing result refers to the parsing result obtained after semantic analysis of the operation. For operations that do not modify user data (the second operation), the processing result refers to the operation context information obtained after processing the operation. The specific implementation of this caching strategy includes the following steps:
[0113] Step 1: Cache Query
[0114] When the audit twin receives an SQL request, it first calculates a query key for that request. This query key can be a hash value of the SQL statement text itself, or a normalized (e.g., removing extra spaces and standardizing parameter formats) hash value of the statement, so that requests with the same semantics can be mapped to the same key. Subsequently, the audit twin uses this query key to perform a lookup in the established cache.
[0115] Step 2: Branch Processing
[0116] Based on the results of the cached query, the following different processing branches will be entered:
[0117] Cache hit: If a processing result associated with the current query key is found in the cache (i.e., a cached parsing result or operation context information), the audit twin will directly use this cached processing result. This means combining the semantic information contained within (such as the tables, fields, and conditions involved in the operation) with the real-time metadata of this request (such as a precise timestamp and the identity of the database user who submitted the request) to generate a new, complete audit record. This process skips the steps of repeatedly parsing, semantically analyzing, and constructing the context for the SQL request.
[0118] Cache miss: If the corresponding processing result is not found in the cache, the audit twin will initiate a complete processing flow. Depending on whether the SQL request is the first or second operation, the audit twin will perform the corresponding parsing or processing operation according to the method defined in the foregoing embodiments of this application, thereby obtaining a completely new parsing result or operation context information.
[0119] Then, the audit twin generates an audit log based on this newly acquired result. Finally, this newly generated processing result is associated with the query key of the current SQL request and stored as a new entry in the cache for use by subsequent identical requests.
[0120] By implementing the above caching strategy, in real-world application scenarios with a large number of repetitive or patterned SQL requests, it is possible to effectively avoid redundant parsing and computation of requests with the same semantics. By directly reusing the intermediate semantic analysis results already cached locally, the audit twin library can reduce the consumption of CPU computing resources and the redundant occupation of memory, thereby improving the throughput and response speed of the audit system under high concurrency loads.
[0121] The electronic device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0122] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.
[0123] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.
[0124] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.
[0125] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0127] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0128] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.
[0129] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0130] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.
[0131] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0132] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.
[0133] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.
[0134] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0135] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A distributed database auditing method, characterized in that, An audit twin is applied to an audit twin database, which maintains metadata synchronization with the distributed database but does not store user data from the distributed database. The metadata includes the table structure, index definitions, and sharding rules of the distributed database. The method includes: Receive an SQL request from the distributed database computing node, the SQL request being used to perform a first operation or a second operation, the first operation being an operation to modify user data, and the second operation being an operation not to modify user data; Perform the following operations based on the metadata synchronized with the distributed database: If the SQL request is the first operation, the semantic scope of the first operation is parsed without actually performing data changes, and an audit record is generated based on the parsing result; If the SQL request is a second operation, process the second operation to obtain operation context information, and generate an audit record based on the operation context information.
2. The method according to claim 1, characterized in that, The audit twin maintains metadata synchronization with the distributed database in the following ways: Meta-information is obtained from the distributed database during the initialization of the audit twin database; When the metadata of the distributed database changes, the metadata of the audit twin database is updated synchronously.
3. The method according to claim 1, characterized in that, The semantic impact of parsing the first operation includes: Identify the data tables and fields involved in the first operation; Parse the conditional constraints of the first operation; Logical estimation of the scope of influence is performed based on the aforementioned constraints.
4. The method according to any one of claims 1-3, characterized in that, The second operation includes at least one of Data Query Language (DQL) operation, Data Definition Language (DDL) operation, and Data Control Language (DCL) operation.
5. The method according to claim 1, characterized in that, The method further includes: During the parsing of the first operation or the processing of the second operation, it is determined whether the SQL request is a cross-shard operation based on the sharding rules. In response to the SQL request being a cross-shard operation, the cross-shard operation information is identified in the generated audit log.
6. The method according to claim 5, characterized in that, The audit log includes at least one of the following: operation type, operation time, operation user, and data objects involved.
7. The method according to claim 1, characterized in that, The method further includes: Maintain the cache of processing results; For a received SQL request, check if there is a cached processing result in the cache; If present, an audit record is generated based on the cached processing result; If it does not exist, perform the corresponding processing operation to obtain a new processing result, generate an audit record, and store the new processing result in the cache; The processing result includes the parsing result or the operation context information.
8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.