Improving database audit accuracy and efficiency using structured audit logs

Abstract

This disclosure relates to improving the accuracy and efficiency of database auditing by utilizing structured audit logs. A computer-implemented method, system, and computer program product for improving the accuracy and efficiency of auditing a database are provided. Tables, lists, or indexes of the database are analyzed to identify metadata, including time-series data, user data, Internet Protocol addresses, and operational data. The identified metadata is associated with corresponding records or rows of the tables, lists, or indexes from which the metadata was extracted. Then, based on the corresponding data operations, it is determined whether to record the original data image associated with the records or rows of the analyzed tables, lists, or indexes. The identified metadata and the recorded data image (if any) are stored in a structured audit log. Audit information is then obtained from the structured audit log by matching the record or row identifier (RID) associated with the query with the RID associated with the structured audit log.

Description

Technical Field

[0001] This disclosure generally relates to database auditing, and more specifically to using structured audit logs to improve the accuracy and efficiency of database auditing. Background Technology

[0002] Database auditing is the monitoring and recording of selected user database actions. It can be based on individual actions, such as the type of SQL statement executed, or a combination of factors such as username, application, and time.

[0003] This type of auditing is typically used to achieve future measurability of current actions taken or affecting specific content within a particular schema, table, or row. Furthermore, such auditing can be used to prevent users (or others) from taking inappropriate actions based on this measurability. Additionally, such auditing enables the investigation of suspicious activity. For example, if a user is deleting data from a table, a security administrator might decide to audit all connections to the database and all successful and unsuccessful deletions of rows from all tables in the database. Moreover, such auditing can be used to monitor and collect data about specific database activity. For example, statistics can be collected on which tables are being updated, how many logical input / output operations are being performed, or how many concurrent users are connecting at peak times. Furthermore, such auditing can be used to detect problems with authorization or access control implementations.

[0004] Based on this type of use of database auditing, it is important to identify which rows and columns of the database, and by whom and when, are accessed. Unfortunately, such information is not easy to track and log, especially for NoSQL (“Unstructured Query Language (SQL)”) databases.

[0005] NoSQL databases provide mechanisms for storing and retrieving data, modeled using methods other than the table relationships used in relational databases. The data structures used by NoSQL databases (e.g., key-value pairs) differ from the default data structures used in relational databases, making some operations faster in NoSQL. Furthermore, NoSQL databases utilize unstructured storage, allowing for high-performance, flexible information processing at scale. For example, NoSQL databases can store unstructured data across multiple processing nodes and servers. However, storing unstructured data across multiple processing nodes and servers makes auditing such databases challenging.

[0006] Currently, auditing plugins can be used to attempt to monitor and log database actions, such as those for NoSQL databases. Unfortunately, they only log data operations (e.g., update and read operations). As a result of monitoring and logging limited data, auditing is deficient in identifying which rows and columns of the database are accessed, by whom, and when.

[0007] Alternatively, schedules can be used to attempt to monitor and log database actions, such as for NoSQL databases. Unfortunately, some operations are not logged. In scenarios using auditing plugins, the use of schedules results in limited data being monitored and logged, leading to deficiencies in auditing's ability to identify which rows and columns in the database, and by whom and when, were accessed. Furthermore, the use of schedules is time-consuming and inefficient, especially when dealing with frequent read, add, delete, and update operations on the database. Additionally, requests for specific information (such as dirty reads) in transactional concurrency (where two transactions run together, such as accessing the same database rows during overlapping time periods) may not be possible using schedules.

[0008] Therefore, there is currently no efficient means to audit database actions in a way that provides accurate proof of data access to the database (e.g., NoSQL database). Summary of the Invention

[0009] In one embodiment of this disclosure, a computer-implemented method for improving the accuracy and efficiency of database auditing includes analyzing tables, lists, or indexes of a database to identify metadata, said metadata including time-series data, user data, Internet Protocol (IP) addresses, and operational data. The method further includes associating the identified metadata with corresponding records or rows of the analyzed tables, lists, or indexes of the database. The method also includes determining, based on corresponding data operations, whether to record a data image associated with a record or row of the analyzed tables, lists, or indexes of the database. Furthermore, the method includes storing the identified metadata and the recorded data image (if any) in a structured audit log, said structured audit log being associated with a record or row identifier of a record or row of the analyzed tables, lists, or indexes of the database.

[0010] In this way, the accuracy and efficiency of database auditing can be improved by utilizing structured audit logs.

[0011] In another embodiment of this disclosure, a computer program product for improving the accuracy and efficiency of database auditing is provided. The computer program product includes one or more computer-readable storage media having program code implemented therewith. The program code includes program instructions for analyzing tables, lists, or indexes of a database to identify metadata, wherein the metadata includes time-series data, user data, Internet Protocol (IP) addresses, and operational data. The program code further includes programming instructions for associating the identified metadata with corresponding records or rows of the analyzed tables, lists, or indexes of the database. The program code also includes programming instructions for determining, based on corresponding data operations, whether to record a data image associated with a record or row of the analyzed tables, lists, or indexes of the database. Furthermore, the program code includes programming instructions for storing the identified metadata and the recorded data image (if any) in a structured audit log, the structured audit log being associated with a record or row identifier of a record or row of the analyzed tables, lists, or indexes of the database.

[0012] In this way, the accuracy and efficiency of database auditing are improved by utilizing structured audit logs.

[0013] In another embodiment of this disclosure, a system includes a memory for storing a computer program for improving the accuracy and efficiency of database auditing, and a processor connected to the memory. The processor is configured to execute program instructions of the computer program, including analyzing tables, lists, or indexes of a database to identify metadata, wherein the metadata includes time-series data, user data, Internet Protocol (IP) addresses, and operational data. The processor is further configured to execute program instructions of the computer program, including associating the identified metadata with corresponding records or rows of the analyzed tables, lists, or indexes of the database. The processor is additionally configured to execute program instructions of the computer program, including determining whether to record a data image associated with a record or row of the analyzed tables, lists, or indexes of the database based on corresponding data operations. Furthermore, the processor is configured to execute program instructions of the computer program, including storing the identified metadata and the recorded data image (if any) in a structured audit log, the structured audit log being associated with a record or row identifier of a record or row of the analyzed tables, lists, or indexes of the database.

[0014] In this way, the accuracy and efficiency of database auditing are improved by utilizing structured audit logs.

[0015] The foregoing has provided a fairly general overview of the features and technical advantages of one or more embodiments of this disclosure in order to better understand the following detailed description of this disclosure. Additional features and advantages of this disclosure will be described below, which form the subject matter of the claims of this disclosure. Attached Figure Description

[0016] A better understanding of this disclosure can be obtained by considering the following detailed description in conjunction with the accompanying drawings, wherein:

[0017] Figure 1 A communication system for implementing the principles of this disclosure is shown according to embodiments thereof;

[0018] Figure 2 This is a diagram of a software component of a database management system that improves the accuracy and efficiency of database auditing by utilizing structured audit logs, according to embodiments of the present disclosure.

[0019] Figure 3 An embodiment of the present disclosure illustrates the hardware configuration of a database management system, which represents a hardware environment for implementing the present disclosure;

[0020] Figure 4 This is a flowchart of a method for establishing a structured audit log according to embodiments of the present disclosure;

[0021] Figure 5 A structured audit log relating to update operations is shown according to an embodiment of this disclosure;

[0022] Figure 6 A structured audit log relating to read or query operations is shown according to an embodiment of this disclosure;

[0023] Figure 7 A structured audit log involving Data Definition Language (DDL) operations is illustrated according to an embodiment of this disclosure;

[0024] Figure 8 A structured audit log relating to Batch Data Manipulation Language (DML) operations is illustrated according to an embodiment of this disclosure;

[0025] Figure 9 This is a flowchart of a method for improving the accuracy and efficiency of database auditing by utilizing structured audit logs, according to embodiments of the present disclosure; and

[0026] Figure 10 The present disclosure illustrates matching and non-matching of record or row identifiers (RIDs) associated with structured audit logs with RIDs associated with queries used to audit the database, according to embodiments of the present disclosure. Detailed Implementation

[0027] As described in the background section, there is currently no means to efficiently audit database actions in a way that allows for accurate verification of data access to a database (e.g., a NoSQL database).

[0028] Embodiments of this disclosure provide a means for establishing and using structured audit logs to accurately demonstrate data access to a database (e.g., a NoSQL database). A more detailed description of these and other features is provided below.

[0029] In some embodiments of this disclosure, the disclosure includes computer-implemented methods, systems, and computer program products for improving the accuracy and efficiency of auditing databases. In one embodiment of this disclosure, tables, lists, or indexes of a database are analyzed to identify metadata, which includes time-series data, user data, Internet Protocol (IP) addresses, and operational data. As used herein, “metadata” refers to data that provides information about other data. As used herein, “time-series data” refers to a specific sequence in which queries are issued to update, insert, delete, etc., information from tables, lists, or indexes of a database. As used herein, “user data” refers to an identifier of a specific user who issues a query to access records or rows in a table, list, or index of a database. As used herein, “Internet Protocol (IP) address” refers to a unique identifier assigned to a device (such as a database). As used herein, “operational data” refers to operations requested to be performed on a database by a query issued by a user of a computing device. In one embodiment, such operations include update operations, delete operations, read operations, data definition language operations (used to define data structures, such as creating, modifying, deleting, and creating views in Structured Query Language), and bulk data manipulation language operations (used to manipulate multiple records or rows of a table, list, or index in a database). The identified metadata is associated with the corresponding record or row of the table, list, or index from which the metadata was extracted. Then, based on the corresponding data operation, it is determined whether to record the raw data image associated with the record or row of the table, list, or index of the analyzed database. As used herein, “raw data image” or “data image” refers to an image of the data resulting from a data operation (e.g., an update operation). For example, a data image of the data obtained from a data operation may be recorded in response to an update or insert operation. However, unless concurrent transactions are present, a data image of the data obtained from a data operation may not be recorded in response to a delete operation or a read / query operation. The identified metadata and the recorded data images (if any) are then stored in a “structured audit log” associated with record or row identifiers of records or rows in tables, lists, or indexes on which the database on which this structured information is obtained. As used herein, a “structured audit log” refers to a data structure that stores the audit information (e.g., metadata, data images of records, etc.). In one embodiment, the structured audit log is stored in a database management system’s storage device (e.g., memory, disk unit), where future queries auditing the database can access such a stored structured audit log to obtain audit information about the database. In one embodiment, appropriate audit information is obtained from the stored structured audit log by matching the row or record identifier associated with the query with the record or row identifier associated with the structured audit log containing the requested database audit information.In this way, the accuracy and efficiency of database auditing can be improved by utilizing structured audit logs.

[0030] In the following description, numerous specific details are set forth in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure can be practiced without such specific details. In other instances, well-known circuits are shown in block diagram form so as not to obscure this disclosure with unnecessary detail. For most, details considering timing considerations, etc., have been omitted because such details are not necessary for obtaining a complete understanding of this disclosure and are within the skill of those skilled in the art.

[0031] Please refer to the attached diagram for details. Figure 1 An embodiment of this disclosure is shown, illustrating a communication system 100 for implementing the principles of this disclosure. The communication system 100 includes a computing device 101 connected to a database management system 102 via a network 103. Furthermore, as... Figure 1 As shown, the database management system 102 is connected to the database 104 (e.g., a NoSQL database).

[0032] Computing device 101 can be any type of computing device (e.g., portable computing unit, personal digital assistant (PDA), laptop computer, mobile device, tablet PC, smartphone, mobile phone, navigation device, gaming unit, desktop computer system, workstation, internet-connected appliance, etc.) configured to connect to network 103 and thus communicate with other computing devices 101 and database management system 102. Note that both computing device 101 and the user of computing device 101 can be identified by element number 101.

[0033] Network 103 can be, for example, a local area network (LAN), a wide area network (WAN), a wireless wide area network (WAN), a circuit-switched telephone network, a Global System for Mobile Communications (GSM) network, a Wireless Application Protocol (WAP) network, a WiFi network, an IEEE 802.11 standard network, or various combinations thereof. Other networks (described herein for brevity) may also be combined without departing from the scope of this disclosure. Figure 1 The system is 100% usable.

[0034] In one embodiment, a user of computing device 101 sends a query to database management system 102 (e.g., a Structured Query Language (SQL) query, a JSON query, etc.). Object representation (objects) are used to update, insert, and delete information from database 104. For example, a user can issue an INSERT INTO query to add new rows of data to a table in database 104. Such queries will be processed by the database management system 102, such as storing and retrieving the data requested by the user.

[0035] In one embodiment, the database management system 102 is configured to maintain a database 104, such as a relational database or a NoSQL database. In one embodiment, the database management system 102 corresponds to an SQL server configured to use Structured Query Language (SQL) to query and maintain the database 104. In another embodiment, the database management system 102 corresponds to a system configured to query JSON stored in the NoSQL database 104. A server that stores object representation data (e.g., JSON objects).

[0036] In one embodiment, as discussed further below, the database management system 102 is configured to build and use structured audit logs for accurate proof of data access to the database 104. In one embodiment, such structured audit logs, if applicable, include metadata (e.g., time-series data, user data, Internet Protocol (IP) addresses, and operational data) as well as a raw data image.

[0037] As used herein, "time series data" refers to a specific order in which queries are issued to update, insert, delete, etc., information from tables, lists, or indexes in database 104. For example, the first query accessing a row may have time series data TS1 (time series #1), and a subsequent query accessing the same row may have time series data TS2, thus indicating that the query follows the query associated with TS1.

[0038] As used herein, “user data” refers to the identifier of a specific user who issues a query to access records or rows of tables, lists, or indexes in database 104.

[0039] As used herein, an "Internet Protocol (IP) address" refers to a unique identifier assigned to a device (such as database 104). In one embodiment, the IP address included in the metadata corresponds to the IP address of database 104 that is being queried by a user of computing device 101.

[0040] As used herein, "operational data" refers to operations performed on database 104 by a query request issued by a user of computing device 101. In one embodiment, such operations include update operations, delete operations, read operations, data definition language operations (used to define data structures, such as creating tables, modifying tables, deleting tables, creating views, etc. in Structured Query Language), and batch data manipulation language operations (used to manipulate multiple records or rows of tables, lists, or indexes in database 104).

[0041] As used herein, "raw data image" refers to an image of data produced by data operations (e.g., update operations). In one embodiment, such an image may include only the data of the changed columns or rows.

[0042] In one embodiment, this information (e.g., metadata, raw data image) is associated with a record or row identifier of a table, list, or index of database 104, wherein such identifier identifies the record or row on which data operations are performed.

[0043] In one embodiment, this information (e.g., metadata, raw data image) is stored by the database management system 102 in a “structured audit log” along with record or row identifiers of tables, lists, or indexes of database 104. As used herein, a “structured audit log” refers to the data structure that stores the information.

[0044] In one embodiment, the structured audit log further includes a pointer to a previously linked query that was executed prior to queries associated with the metadata and raw data image in question (if applicable).

[0045] In one embodiment, the structured audit log is stored in the storage device (e.g., memory, disk unit) of the database management system 102.

[0046] After establishing the structured audit log, the database management system 102 uses the structured audit log for accurate verification of data access in the database 104.

[0047] The following provides a more detailed description of these and other features. Furthermore, the following text combines... Figure 2 Descriptions of the software components of the database management system 102 are provided below, and are combined with Figure 3 Further description of the hardware configuration of the database management system 102 is provided.

[0048] The scope of system 100 is not limited to any particular network architecture. System 100 may include any number of computing devices 101, database management system 102, network 103, and database 104.

[0049] The following combination Figure 2 This provides a discussion of software components used by the database management system 102 to improve the accuracy and efficiency of database auditing by utilizing structured audit logs.

[0050] Figure 2 This is a diagram of software components of a database management system 102 that improves the accuracy and efficiency of database auditing by utilizing structured audit logs, according to embodiments of the present disclosure.

[0051] Combination Figure 1 See Figure 2 The database management system 102 includes an analysis engine 201 configured to analyze tables, lists, and / or indexes of the database 104 to identify metadata, including time-series data, user data, Internet Protocol (IP) addresses, and operational data.

[0052] As used here, "table" refers to a collection of related data stored in a tabular format within database 104.

[0053] As used herein, “list” refers to a set of entries or values, such as entries or values ​​stored in a field of database 104.

[0054] As used herein, "index" refers to a data structure that improves the speed of data retrieval operations on database tables in database 104. An index is used to quickly locate data without having to search every row in the database table every time it is accessed. In one embodiment, an index is created using one or more columns of the database table, providing a basis for efficient access to ordered records and fast random lookups. In one embodiment, an index is a copy of data columns selected from the table, designed to enable efficient searching. In one embodiment, an index includes a "key" or a direct link to the original data row from which the index is copied, allowing for efficient retrieval of complete rows.

[0055] As used herein, “metadata” refers to data that provides information about other data. In one embodiment, metadata includes time-series data, user data, Internet Protocol (IP) addresses, and operational data.

[0056] As described above, "time series data" as used herein refers to the specific order in which queries are issued to update, insert, delete, etc., information from tables, lists, or indexes in database 104. For example, the first query accessing a row may have time series data TS1 (time series #1), and a subsequent query accessing the same row may have time series data TS2, thus indicating that the query follows the query associated with TS1.

[0057] As used herein, “user data” refers to the identifier of a specific user who issues a query to access records or rows of tables, lists, or indexes in database 104.

[0058] As used herein, an "Internet Protocol (IP) address" refers to a unique identifier assigned to a device (such as database 104). In one embodiment, the IP address included in the metadata corresponds to the IP address of database 104 that is being queried by a user of computing device 101.

[0059] As used herein, "operational data" refers to operations performed on database 104 by a query request issued by a user of computing device 101. In one embodiment, such operations include update operations, delete operations, read operations, data definition language operations (used to define data structures, such as creating tables, modifying tables, deleting tables, creating views, etc. in Structured Query Language), and batch data manipulation language operations (used to manipulate multiple records or rows of tables, lists, or indexes in database 104).

[0060] In one embodiment, the analytics engine 201 identifies metadata from tables, lists, or indexes in the database 104 via method calls (such as using the getMetaData() method of a connection object).

[0061] In one embodiment, the analysis engine 201 utilizes a database documentation tool to extract metadata from tables, lists, or indexes in the database 104. Examples of database documentation tools for extracting metadata from tables, lists, or indexes in the database 104 include, but are not limited to, dbdocs.io, Dataedo, etc. SQL SQL, SchemaSpy, dbForgeDocumente, DBScribe, DOC xPress, Innovasys DocumentX, etc.

[0062] The analytics engine 201 is also configured to associate identified metadata with corresponding records or rows in tables, lists, and / or indexes from which the metadata was extracted. In one embodiment, this association is through a record or row identifier (“RID”) that identifies a specific record or row in a table, list, or index. In one embodiment, the record or row identifier (“RID”) corresponds to the address of a row in a table, list, or index.

[0063] For example, if the metadata is extracted from a record or row in a table of database 104 that corresponds to a record or row identifier (“RID”) that is 234567, then the metadata extracted from such a record or row is associated with RID 234567.

[0064] In one embodiment, the analysis engine 201 identifies the RID via a ROWID statement, such as using SQL.

[0065] In one embodiment, the analytics engine 201 is also configured to link queries on records or rows of tables, lists, or indexes of the database 104 to previously executed queries (if any) based on time-series data. For example, a query issued by a user of computing device 101 requesting access to data (such as manipulating or retrieving data) from a specific record or row of a table, list, or index of the database 104 may occur at time T3. A prior query accessing the same record or row of a table, list, or index of the database 104 may occur at time T2, which can occur after the query accessed the same record or row of a table, list, or index of the database 104 at time T1. The query that occurs at time T3 can then be linked to queries that occurred at times T2 and T1.

[0066] In one embodiment, such a link can be established via pointers provided by the analytics engine 201, such as pointers in the structured audit log associated with a query at time T3, which in turn points to the structured audit log associated with a query at time T2, and which contains pointers to the structured audit log associated with a query at time T1. In this way, the result set of historical queries (structured audit logs with metadata and data mappings, if applicable) can be easily retrieved.

[0067] In one embodiment, analytics engine 201 determines such links based on record or row identifiers (“RIDs”) associated with queries. For example, queries occurring at times T1, T2, and T3 may all be associated with the same RID (such as 12345). In one embodiment, analytics engine 201 stores, for example, the RIDs of the analyzed tables, lists, or indexes of database 104 in a data structure (e.g., a table). In one embodiment, analytics engine 201 also stores time-series data associated with the RIDs in that data structure. Such time-series data can be used to link structured audit logs (containing time-series data discussed further below) associated with queries that occurred in the past and were associated with the same RIDs. In one embodiment, such a data structure is stored in the storage device (e.g., memory, disk unit) of database management system 102.

[0068] The database management system 201 further includes a recording engine 202 configured to record raw data images of data produced by data operations (e.g., update operations), such as data produced by an update operation performed on a specific row of a table in database 104. As used herein, "raw data image" refers to an image of data produced by data operations (e.g., update operations).

[0069] In one embodiment, the recording engine 202 determines whether to record a data image of the data obtained from the data operation based on the type of data operation.

[0070] In one embodiment, the recording engine 202 determines the operation type (e.g., update, delete, insert, read, data definition language, bulk data manipulation language, etc.) from the extracted metadata. For example, the analysis engine 201 may have extracted metadata from a table row (identified by RID 12345) containing time series (TS) data TS3, user data "user1", IP address "IP1" of database 104, and data operation "UPD" representing an update operation.

[0071] As discussed above, in one embodiment, the recording engine 202 determines whether to record the raw data image associated with a record or row of an analyzed table, list, or index in database 104 based on the corresponding data operation. In one embodiment, the recording engine 202 records the data image of data resulting from an update (“UPD”) or insert (“INS”) operation after performing the operation. In one embodiment, the recording engine 202 does not record the data image of data resulting from a read / query (“QRY”) operation, except when concurrent transactions exist. In this way, dirty read data involving transaction concurrency can be obtained, as discussed further below. A “dirty read” as used herein occurs when a transaction reads data that has not yet been committed. For example, suppose transaction 1 updates a row in database 104. Transaction 2 reads the updated row before transaction 1 commits the update. This situation is considered to correspond to a “dirty read”.

[0072] In one embodiment, the recording engine 202 does not record a data image of the data generated by the deletion operation (“DLT”).

[0073] In one embodiment, the recording engine 202 records SQL Data Definition Language (DDL) statements as data images generated by DDL operations. In one embodiment, "DDL" operations are used to define data structures, such as creating tables, modifying tables, deleting tables, creating views, etc. in Structured Query Language (SQL).

[0074] In one embodiment, the recording engine 202 records SQL expressions as data images generated by bulk data manipulation language (DML) operations. In one embodiment, DML operations are used to manipulate the data itself. Examples of DML operations include insert, update, or delete instructions using SQL.

[0075] If applicable, examples of software tools used by the recording engine 202 to record data images include, but are not limited to, those that... Hevo Data, HVR wait.

[0076] The database management system 102 further includes an audit log creator 203 configured to create structured audit logs based on information obtained from the analytics engine 201 and the logging engine 202. As used herein, a “structured audit log” refers to a data structure that stores metadata extracted and identified by the analytics engine 201, a raw data image (if any) recorded by the logging engine 202, and any pointers provided by the analytics engine 201 to the structured audit logs associated with historical queries. In one embodiment, such structured audit logs are stored in a storage device (e.g., memory, disk unit) of the database management system 102.

[0077] In one embodiment, the audit log creator 203 stores metadata extracted and identified by the analytics engine 201, raw data images (if any) recorded by the logging engine 202, and any pointers provided by the analytics engine 201 to structured audit logs associated with historical queries in the created structured audit log.

[0078] Examples of software tools used by the audit log creator 203 to create structured audit logs and store the information discussed above in such created structured audit logs include, but are not limited to, those used by the audit log creator 203. Data Modeler, ER / DbSchema, ERBuilder, HeidiSQL, Data Modeler, Toad Data Modeler, Archi, etc.

[0079] The database management system 102 further includes a query engine 204 configured to receive and process queries issued by users of the computing device 101, including queries for auditing the database 104 via structured audit logs.

[0080] In one embodiment, query engine 204 (e.g., Drill, Impala, Spark (or similar devices) is configured to receive query requests from a user of computing device 101 for auditing database 104. In one embodiment, the query for auditing database 104 includes record or row identifiers (RIDs) of records or rows in tables, lists, or indexes of database 104 to which the audit is performed. In one embodiment, query engine 204 searches for RIDs associated with structured audit logs to determine if a match exists. In one embodiment, each structured audit log is associated with a RID, where such information is stored in a data structure. In one embodiment, query engine 204 searches such a data structure to identify any matching RIDs. In one embodiment, query engine 204 utilizes natural language processing to identify any matches in the data structure. In one embodiment, such a data structure is stored in a storage device (e.g., memory, disk unit) of database management system 102.

[0081] In one embodiment, query engine 204 analyzes query requests from users of computing device 101 for auditing database 104 to determine which records or rows in tables, lists, or indexes of database 104 should be audited. For example, the query may include a record or row number function or select clause that identifies records or rows in tables, lists, or indexes of database 104 for auditing. In identifying records or rows in tables, lists, or indexes of database 104, record or row identifiers (RIDs) may be identified via a ROWID statement.

[0082] Once a RID is identified, query engine 204 can determine whether a match exists between such a RID and the RID of the structured audit log stored in the aforementioned data structure. For example, as discussed above, in one embodiment, query engine 204 utilizes natural language processing to identify any matches in the data structure.

[0083] If a match exists between the RID associated with the query and the RID of the structured audit log, the query engine 204 retrieves audited information (e.g., metadata, raw data image) from the matching structured audit log. This audited information is then provided to the user of the computing device 101, for example, via the user interface of the computing device 101.

[0084] Alternatively, if there is no match between the RID associated with the query and the RID of the structured audit log, the query engine 204 notifies the requester (the user of computing device 101) that the record or row in the table, list, or index of database 104 has not yet been accessed.

[0085] The following section provides further description of these and other features in conjunction with a discussion of methods for improving the accuracy and efficiency of database auditing by leveraging structured audit logs.

[0086] Before discussing methods to improve the accuracy and efficiency of database auditing by utilizing structured audit logs, the following will combine... Figure 3 Provide database management system 102 ( Figure 1 Description of the hardware configuration.

[0087] See now Figure 3 , Figure 3 The diagram shows a database management system 102 representing a hardware environment for implementing this disclosure. Figure 1 The hardware configuration of this disclosure is an embodiment of the present disclosure.

[0088] The database management system 102 has a processor 301 connected to various other components via a system bus 302. The operating system 303 runs on the processor 301 and provides access to... Figure 3 The application 304 controls and coordinates the functions of different components according to the principles of this disclosure. It operates in conjunction with the operating system 303 and provides calls to the operating system 303, wherein the calls implement different functions or services to be performed by the application 304. The application 304 may include, for example, an analysis engine 201. Figure 2 Recording Engine 202 Figure 2 ), Audit Log Creator 203 ( Figure 2 ) and query engine 204 ( Figure 2 Furthermore, application 304 may include, for example, procedures for improving the accuracy and efficiency of database auditing by leveraging structured audit logs, as combined with the following Figure 4-10 Further discussion is needed.

[0089] See you again Figure 3 Read-only memory (“ROM”) 305 is connected to system bus 302 and includes a basic input / output system (“BIOS”) that controls certain basic functions of the database management system 102. Random access memory (“RAM”) 306 and disk adapter 307 are also connected to system bus 302. It should be noted that software components, including operating system 303 and applications 304, can be loaded into RAM 306 for execution, which may serve as the main memory of the database management system 102. Disk adapter 307 may be an integrated drive electronics (“IDE”) adapter that communicates with disk unit 308 (e.g., a disk drive). Note that, as combined below... Figure 4-10 Further discussion suggests that procedures for improving the accuracy and efficiency of database auditing by utilizing structured audit logs may reside in disk unit 308 or application 304.

[0090] The database management system 102 may further include a communication adapter 309 connected to the bus 302. The communication adapter 309 connects the bus 302 to an external network (e.g., Figure 1 Network 103) interconnects with devices such as computing device 101 ( Figure 1 ) communicates with other devices.

[0091] In one embodiment, application 304 of database management system 102 includes software components such as analysis engine 201, logging engine 202, audit log creator 203, and query engine 204. In one embodiment, such components may be implemented in hardware, wherein such hardware components are connected to bus 302. The functions performed by these components discussed above are not general-purpose computer functions. Therefore, database management system 102 is a specific machine as a result of implementing specific, non-general-purpose computer functions.

[0092] In one embodiment, the functionality of such software components of the database management system 102 (e.g., analysis engine 201, logging engine 202, audit log creator 203, and query engine 204) includes features for improving the accuracy and efficiency of database auditing by leveraging structured audit logs, which may be implemented in an application-specific integrated circuit (ASIC).

[0093] The present invention can be a system, method, and / or computer program product at any possible level of technical detail in the integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0094] Computer-readable storage media can be tangible devices capable of retaining and storing instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0095] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0096] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0097] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0098] These computer-readable program instructions may be provided to a computer processor or other programmable data processing apparatus to produce a machine, such that these instructions, which execute via the computer processor or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, thereby comprising an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0099] These computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, thereby causing the instructions to be executed on the computer, other programmable apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0101] As mentioned above, database auditing is the monitoring and logging of selected user database actions. It can be based on individual actions, such as the type of SQL statement executed, or a combination of factors such as username, application, and time. Based on the use of database auditing, identifying which rows and columns of the database, and by whom and when, is important. Unfortunately, such information is not easily tracked and logged, especially for NoSQL (“Unstructured Query Language (SQL)”) databases. NoSQL databases provide mechanisms for storing and retrieving data that are modeled in a manner different from the table relationships used in relational databases. The data structures used by NoSQL databases (e.g., key-value pairs) differ from the default data structures used in relational databases, making some operations faster in NoSQL. Furthermore, NoSQL databases utilize unstructured storage, which allows for high-performance, flexible information processing at scale. For example, NoSQL databases can store unstructured data across multiple processing nodes and across multiple servers. However, storing unstructured data across multiple processing nodes and across multiple servers makes auditing such databases challenging. Currently, auditing plugins can be used to attempt to monitor and log database actions, such as those for NoSQL databases. Unfortunately, only data operations (e.g., update and read operations) are logged. As a result of monitoring and logging limited data, auditing is deficient in identifying which rows and columns of the database are accessed, by whom, and when. Alternatively, schedules can be used to attempt to monitor and log database actions, such as for NoSQL databases. Unfortunately, some operations are not logged. In scenarios using auditing plugins, the use of schedules results in limited data being monitored and logged, leading to a deficiency in auditing in identifying which rows and columns of the database, by whom, and when. Furthermore, the use of schedules is time-consuming and inefficient, especially involving frequent read, add, delete, and update operations on the database. Additionally, requests for specific information (such as dirty reads) in transactional concurrency (two transactions running together, such as accessing the same database rows during overlapping time periods) may not be possible using schedules. Consequently, there is currently no means to efficiently audit database actions in a way that allows for accurate proof of data access to a database (e.g., a NoSQL database).

[0102] Embodiments of this disclosure provide a means for constructing and using structured audit logs for accurate proof of data access in a database (e.g., a NoSQL database), as described below. Figure 4-10 The subject of discussion. Figure 4 This is a flowchart of a method for creating structured audit logs. Figure 5 The structured audit logs involving update operations are shown. Figure 6The structured audit logs involving read or query operations are shown. Figure 7 The structured audit logs involving Data Definition Language (DDL) operations are shown. Figure 8 The structured audit logs involving Batch Data Manipulation Language (DML) operations are shown. Figure 9 This is a flowchart of a method for improving the accuracy and efficiency of database auditing by utilizing structured audit logs. Figure 10 This shows the matching and non-matching of record or row identifiers (RIDs) associated with structured audit logs with RIDs associated with queries in the audit database.

[0103] As mentioned above, Figure 4 This is a flowchart of a method 400 for establishing a structured audit log according to an embodiment of the present disclosure.

[0104] Combination Figure 1-3 See Figure 4 In operation 401, the analysis engine 201 of the database management system 102 analyzes the tables, lists, or indexes of the database 104 to identify metadata, which includes time-series data, user data, Internet Protocol (IP) addresses, and operational data.

[0105] As discussed above, the term "table" as used herein refers to a collection of related data stored in a tabular format within database 104.

[0106] As used herein, “list” refers to a set of entries or values, such as entries or values ​​stored in a field of database 104.

[0107] As used herein, "index" refers to a data structure that improves the speed of data retrieval operations on database tables in database 104. An index is used to quickly locate data without having to search every row in the database table every time it is accessed. In one embodiment, an index is created using one or more columns of the database table, providing a basis for efficient access to ordered records and fast random lookups. In one embodiment, an index is a copy of data columns selected from the table, designed to enable efficient searching. In one embodiment, an index includes a "key" or a direct link to the original data row from which the index is copied, allowing for efficient retrieval of complete rows.

[0108] As used herein, “metadata” refers to data that provides information about other data. In one embodiment, metadata includes time-series data, user data, Internet Protocol (IP) addresses, and operational data.

[0109] Furthermore, as mentioned above, "time series data" as used herein refers to the specific order in which queries are issued to update, insert, delete, etc., information from tables, lists, or indexes in database 104. For example, the first query accessing a row may have time series data TS1 (time series #1), and a subsequent query accessing the same row may have time series data TS2, thus indicating that the query follows the query associated with TS1.

[0110] As used herein, “user data” refers to the identifier of a specific user who issues a query to access records or rows of tables, lists, or indexes in database 104.

[0111] As used herein, an "Internet Protocol (IP) address" refers to a unique identifier assigned to a device (such as database 104). In one embodiment, the IP address included in the metadata corresponds to the IP address of database 104 that is being queried by a user of computing device 101.

[0112] As used herein, "operational data" refers to operations performed on database 104 by a query request issued by a user of computing device 101. In one embodiment, such operations include update operations, delete operations, read operations, data definition language operations (used to define data structures, such as creating tables, modifying tables, deleting tables, creating views, etc. in Structured Query Language), and batch data manipulation language operations (used to manipulate multiple records or rows of tables, lists, or indexes in database 104).

[0113] In one embodiment, the analytics engine 201 identifies metadata from tables, lists, or indexes in the database 104 via method calls (such as using the getMetaData() method of a connection object).

[0114] In one embodiment, the analysis engine 201 utilizes a database documentation tool to extract metadata from tables, lists, or indexes in the database 104. Examples of database documentation tools for extracting metadata from tables, lists, or indexes in the database 104 include, but are not limited to, dbdocs.io, Dataedo, etc. SQL SQL, SchemaSpy, dbForgeDocumente, DBScribe, DOC xPress, Innovasys DocumentX, etc.

[0115] In operation 402, the analysis engine 201 of the database management system 102 associates the identified metadata with the corresponding record or row of the table, list, or index from which the metadata was extracted.

[0116] As discussed above, in one embodiment, this association is through a record or row identifier (“RID”) that identifies a specific record or row in a table, list, or index. In one embodiment, the record or row identifier (“RID”) corresponds to the address of the row in the table, list, or index.

[0117] For example, if the metadata is extracted from a record or row in a table of database 104 that corresponds to record or row identifier (“RID”) 234567, then the metadata extracted from such a record or row is associated with RID 234567.

[0118] In one embodiment, the analysis engine 201 identifies the RID via a ROWID statement, such as using SQL.

[0119] In operation 403, the record engine 202 of the database management system 102 determines the type of data operation (e.g., update, delete, insert, read, data definition language, data manipulation language, etc.) from the identified metadata. For example, the analysis engine 201 may have extracted metadata from a table row (identified by RID 12345) containing time series (TS) data TS3, user data "user1", the IP address of database 104 "IP1", and the data operation "UPD" representing an update operation.

[0120] In operation 404, the recording engine 202 of the database management system 102 determines, based on the corresponding data operation, whether to record the original data image associated with the records or rows of the table, list, or index being analyzed.

[0121] If the recording engine 202 determines, based on the corresponding data operation, not to record the original data image associated with the records or rows of the table, list, or index being analyzed, then in operation 405, the recording engine 202 of the database management system 102 does not record the data image associated with the records or rows of the table, list, or index being analyzed. A further explanation of operation 405 is provided below.

[0122] However, if the recording engine 202 determines the original data image associated with the records or rows of the table, list, or index of the analysis based on the corresponding data operation, then in operation 406, the recording engine 202 of the database management system 102 records the data image of the data obtained by the operation, as described below. Figure 5-8 Further discussion is needed.

[0123] As discussed above, the term "raw data image" or "data image" as used herein refers to an image of the data obtained from a data operation (e.g., an update operation).

[0124] In one embodiment, the recording engine 202 determines whether to record a data image of the data generated by the data operation based on the type of data operation.

[0125] In one embodiment, the recording engine 202 determines the operation type (e.g., update, delete, insert, read, data definition language, bulk data manipulation language, etc.) from the extracted metadata. For example, the analysis engine 201 may have extracted metadata from rows (identified by RID 12345) of a table in database 104, which includes time series (TS) data TS3, user data "user1", the IP address of database 104 "IP1", and the data operation "UPD" representing an update operation.

[0126] As described above, in one embodiment, the recording engine 202 determines whether to record the raw data image associated with a record or row of a table, list, or index of the database 104 being analyzed, based on the corresponding data operation. In one embodiment, the recording engine 202 records the data image of the data produced by an update (“UPD”) or insert (“INS”) operation after the operation is performed. In one embodiment, the recording engine 202 does not record the data image of the data produced by a read / query (“QRY”) operation, except when concurrent transactions exist. In this way, dirty read data involving transaction concurrency can be obtained, as discussed further below. A “dirty read” as used herein occurs when a transaction reads data that has not yet been committed. For example, suppose transaction 1 updates a row in database 104. Transaction 2 reads the updated row before transaction 1 commits the update. This situation is considered to correspond to a “dirty read”.

[0127] In one embodiment, the recording engine 202 does not record a data image of the data generated by the deletion operation (“DLT”).

[0128] In one embodiment, the recording engine 202 records SQL Data Definition Language (DDL) statements as data images generated by DDL operations. In one embodiment, "DDL" operations are used to define data structures, such as creating tables, modifying tables, deleting tables, creating views, etc. in Structured Query Language (SQL).

[0129] In one embodiment, the recording engine 202 records SQL expressions as data images generated by bulk data manipulation language (DML) operations. In one embodiment, DML operations are used to manipulate the data itself. Examples of DML operations include insert, update, or delete instructions in SQL.

[0130] If applicable, examples of software tools used by the recording engine 202 to record data images include, but are not limited to, those that... Hevo Data, HVR wait.

[0131] After the recording engine 202 records or does not record the data image of the data generated by data operations, in operation 407, the analysis engine 201 of the database management system 102, based on time-series data, links queries on records or rows of tables, lists, or indexes in the database 104 to previously executed queries (if any) on the records or rows of the analyzed tables, lists, or indexes. For example, a query issued by a user of computing device 101 requesting access to data (such as manipulating or retrieving data) from a specific record or row of a table, list, or index in the database 104 may occur at time T3. A prior query accessing the same record or row of a table, list, or index in the database 104 may occur at time T2, which may occur after the query accessing the same record or row of a table, list, or index in the database 104 at time T1. The query that occurs at time T3 can then be linked to the queries that occurred at times T2 and T1.

[0132] As described above, in one embodiment, such a link can be established via pointers provided by the analytics engine 201, such as pointers in the structured audit log associated with a query at time T3, which in turn points to the structured audit log associated with a query at time T2, and which contains pointers to the structured audit log associated with a query at time T1. In this way, the result set of historical queries (structured audit logs with metadata and data mappings, if applicable) can be easily retrieved.

[0133] In one embodiment, analytics engine 201 determines such links based on record or row identifiers (“RIDs”) associated with queries. For example, queries occurring at times T1, T2, and T3 may all be associated with the same RID (such as 12345). In one embodiment, analytics engine 201 stores, for example, the RIDs of the analyzed tables, lists, or indexes of database 104 in a data structure (e.g., a table). In one embodiment, analytics engine 201 also stores time-series data associated with RIDs in a data structure. Such time-series data can be used to link structured audit logs (containing time-series data discussed further below) associated with queries that occurred in the past and were associated with the same RID. In one embodiment, such a data structure is stored in a storage device (e.g., memory 305, disk unit 308) of database management system 102.

[0134] In operation 408, the audit log creator 203 of the database management system 102 creates a structured audit log and stores audit information within the created structured audit log, such as metadata extracted and identified by the analysis engine 201, the raw data image recorded by the recording engine 202 (if any), and any pointers provided by the analysis engine 201 to the structured audit log associated with historical queries (i.e., pointers to previously executed queries on the same records or rows of the analyzed tables, lists, or indexes of the database 102 associated with the created structured audit log). In one embodiment, such a structured audit log and the audit information stored in the structured audit log are associated with record or row identifiers of the records or rows of the analyzed tables, lists, or indexes.

[0135] As discussed above, "structured audit log" as used herein refers to a data structure that stores metadata extracted and identified by analytics engine 201, raw data images (if any) recorded by logging engine 202, and any pointers provided by analytics engine 201 to structured audit logs associated with historical queries. In one embodiment, such structured audit logs are stored in a storage device (e.g., memory 305, disk unit 308) of database management system 102.

[0136] Examples of software tools used by the audit log creator 203 to create structured audit logs and store the audit information discussed above in such created structured audit logs include, but are not limited to, those used by the audit log creator 203. Data Modeler, ER / DbSchema, ERBuilder, HeidiSQL, Data Modeler, Toad DataModeler, Archi, etc.

[0137] The following combination Figure 5-8 The discussion includes a diagram illustrating the storage of metadata and data images (if any) of records in a structured audit log, along with pointers to any previously executed queries.

[0138] As discussed above, in one embodiment, the recording engine 202 records a data image of the data generated from the update (“UPD”) or insert (“INS”) operation after the operation is performed, such as in Figure 5 As shown in the figure.

[0139] Figure 5 A structured audit log 501 relating to update operations is shown according to an embodiment of the present disclosure.

[0140] As will be discussed in further detail below, the extracted metadata (including data images of any records) will be stored in a structured audit log (such as... Figure 5 In the structured audit log (500).

[0141] like Figure 5 As shown, the structured audit log 501 is associated with a record or row identifier (RID) 502. In one embodiment, the structured audit log 501 includes metadata fields 503 identified by the analysis engine 201 for records or rows of the analyzed tables, lists, or indexes of database 104 identified by RID 502. For example, metadata 503 includes time series (TS) data 504, user data 505, IP addresses 506 (such as the IP address 506 of database 104), and data operations (“OP”) 507. Figure 5 As shown, TS data 504 corresponds to the time of TS3. User data 505 corresponds to user #1 (“USR1”). The IP address 506 of database 104 corresponds to “IP1” (e.g., 192.158.1.38). Data operation 507 corresponds to update operation (“UPD”).

[0142] In addition, in one embodiment, the structured audit log 501 includes a raw data image field 508, which can be used to store a data image of the record.

[0143] In one embodiment, to save storage space, the recording engine 202, as Figure 5 The data image 508 shown records the data generated by the update (“UPD”) operation after the operation is performed. For example, such a data image 508 is recorded as “Img1v3” and stored in the structured audit log 501.

[0144] In addition, such as Figure 5 As shown, structured audit log 501 includes a pointer 509 to structured audit log 510, which is similarly structured to structured audit log 501. As described above, time-series data can be used to link structured audit logs that involve the same RID. For example, structured audit log 501 is linked to structured audit log 510 because both are associated with the same RID, and structured audit log 510 is associated with time T2 (see time-series data field 504 of structured audit log 510) that occurs just before time T3 (see time-series data field 504 of structured audit log 501). Similarly, as... Figure 5As shown, the structured audit log 510 includes a pointer 511 to a structured audit log 512 (structured similarly to the structured audit log 501), which involves the same RID as the structured audit log 501, but is associated with a time T1 (see the time series data field 504 of the structured audit log 512) that occurs just before time T2 (see the time series data field 504 of the structured audit log 510).

[0145] In addition, such as Figure 5 As shown, the recording engine 202 records a data image of the data obtained by the insert (“INS”) operation after the operation is performed (see data operation field 507 of the structured audit log 512). For example, such a data image is recorded and stored in the raw data image field 508 of the structured audit log 512 as “Img1v1”.

[0146] As discussed above, to save storage space, the recording engine 202 records a data image of the data generated by the update (“UPD”) operation after the operation is performed. To obtain a data image before the update operation, a link to the previous image can be used via pointers as described above.

[0147] In one embodiment, the data image of such records may include only the data of the changed rows or columns of tables, lists, or indexes of database 104.

[0148] In one embodiment, the recording engine 202 does not record data images of data generated by read / query operations (“QRY”) except when concurrent transactions exist, such as Figure 6 As shown. In this way, dirty reads in concurrent transactions can be obtained as discussed further below. A "dirty read," as used in this article, occurs when a transaction reads data that has not yet been committed. For example, suppose transaction 1 updates a row in database 104. Transaction 2 reads the updated row before transaction 1 commits the update. This situation is considered to correspond to a "dirty read."

[0149] See Figure 6 , Figure 6 A structured audit log relating to read or query operations is shown according to an embodiment of this disclosure.

[0150] As previously combined Figure 5 The structured audit log 501 discussed here is associated with a record or row identifier (RID) 502. Such a structured audit log 501 can be associated with a pointer to a structured audit log 602 (and...). Figure 5 The structured audit log 501 is similarly structured, and pointers to 601 are associated with it, such as... Figure 6As shown in the diagram. Structured audit log 501 is linked to structured audit log 602 because they are associated with the same RID, and structured audit log 602 is associated with time T2 (see time series data field 504 of structured audit log 602) that occurred just before time T3 (see time series data field 504 of structured audit log 501).

[0151] In one embodiment, the recording engine 202 typically does not record the raw data image of read / query operations (“QRY”). However, when concurrent transactions occur, such as Figure 6 As shown, the recording engine 202 records the resulting images of read / query operations, which are stored in a structured audit log by the audit log creator 203.

[0152] For example, such as Figure 6 As shown, structured audit logs 602, 603, and 604 (all structured similarly to structured audit log 501) are all associated with the time series data TS2 in the time series data field 504. As shown in structured audit log 603, an update operation occurred in database 104, which resulted in the execution of the above-described combination... Figure 5 The data image of the data produced by the discussed update (“UPD”) operation is recorded after the operation (see “Img1v2” in the original data image field 508). However, as shown in the structured audit log 604, a read operation (“QRY”) as shown in the data operation field 507 may be performed before the update operation is committed. Therefore, the structured audit log 602 stores a data image of the record of the read operation (see “Img1v1” in the original data image field 508). In this way, dirty read data involving transaction concurrency can be obtained.

[0153] In addition, such as Figure 6 As shown, structured audit log 602 has a pointer 605 to structured audit log 603, and structured audit log 603 has a pointer 606 to structured audit log 604. Each of these structured audit logs 602, 603, and 604 occurs at time T2. Specifically, structured audit logs 603 and 604 are concurrent transactions because they occur simultaneously. As discussed above, when concurrent transactions occur, such as read / query operations performed before committing update operations, structured audit logs (such as structured audit log 602) are created to store the original data image of the read operations, as explained above.

[0154] In addition, such as Figure 6 As shown, the structured audit log 604 includes a pointer 607 to link to the structured audit log 608 (and...). Figure 5The structured audit log 501 is similarly structured. As mentioned above, time-series data can be used to link structured audit logs that involve the same RID. For example, structured audit log 608 is linked to structured audit log 604 because both are associated with the same RID, and structured audit log 608 is associated with time T1 (see time-series data field 504 of structured audit log 608) that occurs just before time T2 (see time-series data field 504 of structured audit log 604).

[0155] Furthermore, in one embodiment, the recording engine 202 does not record the data image of the data generated by the deletion operation (“DLT”).

[0156] In one embodiment, the recording engine 202 records SQL Data Definition Language (DDL) statements as data images generated by DDL operations, such as... Figure 7 As shown. In one embodiment, "DDL" operations are used to define data structures, such as creating a table, deleting a table, modifying a table, creating a view, etc. in Structured Query Language (SQL).

[0157] Figure 7 A structured audit log 701 involving data definition language (DDL) operations is illustrated according to an embodiment of this disclosure.

[0158] See Figure 7 Structured audit log 701 (and Figure 5 The structured audit log 501 (similarly structured) is created by the audit log creator 203 to store metadata obtained through the analysis engine 201 and a data image recorded by the recording engine 202. For example... Figure 7 As shown, the data operation in data operation field 507 corresponds to "alter table," which is a DDL operation. In this case, record engine 202 records the SQL DDL statement as shown in the original data image field 508.

[0159] In addition, such as Figure 7 As shown, structured audit log 701 is linked to structured audit log 702 via pointer 703. As described above, time-series data can be used to link structured audit logs that involve the same RID. For example, structured audit log 701 is linked to structured audit log 702 because both structured audit log 701 and structured audit log 702 are associated with the same RID, and structured audit log 702 is associated with time T2 (see time-series data field 504 of structured audit log 702) that occurs just before time T3 (see time-series data field 504 of structured audit log 701).

[0160] like Figure 7 As shown, the structured audit log 702 also includes a DDL operation (“createview”) in the data manipulation field 507, which causes the logging engine 202 to record SSL DDL statements as shown in the raw data image field 508 of the structured audit log 702.

[0161] In one embodiment, the recording engine 202 records SQL expressions as data images generated by Batch Data Manipulation Language (DML) operations, such as... Figure 8 As shown. In one embodiment, DML operations are used to manipulate the data itself. Examples of DML operations include insert, update, or delete commands in SQL.

[0162] Figure 8 A structured audit log involving bulk Data Manipulation Language (DML) operations is illustrated according to an embodiment of this disclosure.

[0163] See Figure 8 In one embodiment, the expression is used to track batch operations, such as batch DML operations. For example, a batch DML operation may include updating multiple records.

[0164] like Figure 8 As shown, the structured audit log 801 (and) Figure 5 The structured audit log 501 (similarly structured) is associated with two RIDs 502 (i.e., RID=1802 and RID=2803), which represent multiple records or rows of tables, lists, or indexes in database 104 that are being updated by batch DML operations (multiple update operations). In one embodiment, the audit log creator 203 stores the SQL DDL statement (“Col1*0.21+Col2”) associated with the batch DML operation in the raw data image field 508 of the structured audit log 801.

[0165] In addition, such as Figure 8 As shown, in such batch DML operations, the structured audit logs will show errors 804 and 805 (similar to...). Figure 5 The structured audit log (501 is structured) indicates multiple update operations, such as... Figure 8 As shown, structured audit log 805 is linked to structured audit log 806 via pointer 807 (the structure is similar to...). Figure 5(Structured audit log 501). As mentioned above, time-series data can be used to link structured audit logs that involve the same RID. For example, structured audit log 805 is linked to structured audit log 806 because both are associated with the same RID, and structured audit log 806 is associated with time T1 (see time-series data field 504 of structured audit log 806) that occurs just before time T2 (see time-series data field 504 of structured audit log 805).

[0166] After creating a structured audit log, the structured audit log can be used to audit the database (104 error), as shown in the following combination. Figure 9 The subject of discussion.

[0167] Figure 9 This is a flowchart of a method 900 for improving the accuracy and efficiency of database auditing by utilizing structured audit logs, according to embodiments of the present disclosure.

[0168] Combination Figure 1-8 See Figure 9 In operation 901, the query engine 204 of the database management system 102 (e.g., Drill, Impala, Spark and other similar devices receive query requests from the user of computing device 101 for auditing database 104.

[0169] In operation 902, the query engine 204 of the database management system 102 identifies the record or row identifier (RID) associated with the query request.

[0170] As described above, in one embodiment, a query for auditing database 104 includes record or row identifiers (RIDs) of records or rows in tables, lists, or indexes of database 104 to which the audit is performed. In one embodiment, query engine 204 searches for RIDs associated with structured audit logs to determine if a match exists. In one embodiment, each structured audit log is associated with a RID, wherein such information is stored in a data structure. In one embodiment, query engine 204 searches such a data structure to identify any matching RIDs. In one embodiment, query engine 204 utilizes natural language processing to identify any matches in the data structure. In one embodiment, such a data structure is stored in a storage device (e.g., memory 305, disk unit 308) of database management system 102.

[0171] In one embodiment, query engine 204 analyzes query requests from users of computing device 101 for auditing database 104 to determine which records or rows in tables, lists, or indexes of database 104 should be audited. For example, the query may include a record or row number function or select clause that identifies the records or rows in tables, lists, or indexes of database 104 to be audited. In identifying records or rows in tables, lists, or indexes of database 104, record or row identifiers (RIDs) may be identified via a ROWID statement.

[0172] In operation 903, the query engine 204 of the database management system 102 determines whether there is a match between the record or row identifier (RID) associated with the query and the record or row identifier associated with the structured audit log (such as structured audit log 501).

[0173] As described above, the RID of the structured audit log can be stored in a data structure, such as in the storage device of the database management system 102 (e.g., memory 305, disk unit 308). In one embodiment, the query engine 204 uses natural language processing to identify any matches in such a data structure.

[0174] If a match exists between the RID associated with the query and the RID associated with the structured audit log, then in operation 904, the query engine 204 of the database management system 102 will retrieve the match from the RID associated with the query. Figure 10 The matched structured audit logs shown retrieve audited information. Such audited information may include information from metadata field 503 and any data image from the original data image field 508. Furthermore, such audited information may include any audited information from any linked structured audit logs, as combined below. Figure 10 Further discussion is needed.

[0175] In operation 905, the query engine 204 of the database management system 102, such as via the following combination Figure 10 The user interface of the computing device 101 under discussion provides such audited information to the user of the computing device 101.

[0176] Figure 10 The illustration shows matching and non-matching of record or row identifiers (RIDs) associated with structured audit logs and RIDs associated with queries used to audit the database, according to embodiments of this disclosure.

[0177] See Figure 10Query engine 204 can perform a search for the RID associated with the query in the data structure that stores the RIDs of structured audit logs (such as structured query log 1001). If the query is associated with RID=1, and structured query log 1001 is associated with RID=1 (see...), then... Figure 10 Element 1002), as indicated in the data structure and Figure 10 As shown, query engine 204 will identify the RID of structured query log 1001 as a match for the RID associated with the query. After matching the RID associated with the query with the RID associated with the structured audit log, query engine 204 obtains audited information, such as information in metadata field 503 and data image in original data image field 508 (if any). Furthermore, such audited information may include any linked historical queries, such as audited information in structured audit log 1003 linked to structured audit log 1001 via pointer 1004. Additionally, such audited information may include audited information in structured audit log 1005, which is linked to structured audit log 1003 via pointer 1006, such as... Figure 10 As shown. Such matched structured audit logs are considered to be within the "RID Matching Region" 1007; however, those structured audit logs associated with RIDs that do not match the RIDs associated with the queries used to audit database 104 are referred to as being within the "RID Mismatching Region" 1008.

[0178] For example, structured audit log 1009 may not be associated with the RID of any query received by query engine 204 from audit database 104. Figure 10 As shown, structured audit log 1009 is linked to structured audit log 1010 via pointer 1011.

[0179] return Figure 9 In operation 903, if there is no match between the RID associated with the query used to audit database 104 and the RID of the structured audit log, then in operation 906, the query engine 204 of the database management system 102 notifies the requester (user of computing device 101) via the user interface of computing device 101 that a record or row of a table, list, or index of database 104 has not been accessed.

[0180] As a result of the foregoing, embodiments of this disclosure provide a means for constructing and using structured audit logs to accurately demonstrate data access to a database (e.g., a NoSQL database).

[0181] Furthermore, the principles of this disclosure improve upon techniques or fields involving database auditing. As mentioned above, database auditing is the monitoring and recording of selected user database actions. It can be based on individual actions, such as the type of SQL statement executed, or a combination of factors such as username, application, and time. Based on the use of database auditing, it is important to identify which rows and columns of the database, and by whom and when, are accessed. Unfortunately, such information is not easily tracked and recorded, especially for NoSQL (“Unstructured Query Language (SQL)”) databases. NoSQL databases provide mechanisms for storing and retrieving data that are modeled in a way that differs from the table relationships used in relational databases. The data structures used by NoSQL databases (e.g., key-value pairs) differ from the default data structures used in relational databases, making some operations faster in NoSQL. Furthermore, NoSQL databases utilize unstructured storage, which allows for high-performance, flexible information processing at scale. For example, NoSQL databases can store unstructured data across multiple processing nodes and across multiple servers. However, storing unstructured data across multiple processing nodes and across multiple servers makes auditing such databases challenging. Currently, auditing plugins can be used to attempt to monitor and log database actions, such as those for NoSQL databases. Unfortunately, only data operations (e.g., update and read operations) are logged. As a result of monitoring and logging limited data, auditing is deficient in identifying which rows and columns of the database are accessed, by whom, and when. Alternatively, schedules can be used to attempt to monitor and log database actions, such as those for NoSQL databases. Unfortunately, some operations are not logged. As in scenarios using auditing plugins, the use of schedules results in limited data being monitored and logged, thus making auditing deficient in identifying which rows and columns of the database, by whom, and when. Furthermore, the use of schedules is time-consuming and inefficient, especially involving frequent read, add, delete, and update operations on the database. Additionally, requests for specific information (such as dirty reads) in transactional concurrency (two transactions running together, such as accessing the same database rows during overlapping time periods) may not be possible using schedules. As a result, there is currently no means to efficiently and effectively audit database actions that can provide accurate proof of data access to a database (e.g., a NoSQL database).

[0182] Embodiments of this disclosure improve upon this technique by analyzing tables, lists, or indexes of a database to identify metadata, including time-series data, user data, Internet Protocol (IP) addresses, and operational data. As used herein, “metadata” refers to data that provides information about other data. As used herein, “time-series data” refers to a specific sequence of queries issued to update, insert, delete, etc., information from a table, list, or index of a database. As used herein, “user data” refers to an identifier of a specific user who issues a query to access a record or row in a table, list, or index of a database. As used herein, “Internet Protocol (IP) address” refers to a unique identifier assigned to a device (such as a database). As used herein, “operational data” refers to operations performed on a database by a query request issued by a user of a computing device. In one embodiment, such operations include update operations, delete operations, read operations, data definition language operations (for defining data structures, such as creating a table, modifying a table, deleting a table, creating a view, etc. in Structured Query Language), bulk data manipulation language operations (for manipulating multiple records or rows in a table, list, or index of a database), etc. The identified metadata is associated with the corresponding record or row in the table, list, or index of the database from which the metadata was extracted. Then, based on the corresponding data operation, it is determined whether to record the raw data image associated with the record or row of the table, list, or index of the database being analyzed. As used herein, "raw data image" or "data image" refers to an image of the data produced by a data operation (e.g., an update operation). For example, a data image of the data obtained from a data operation may be recorded in response to an update or insert operation. However, except in the presence of concurrent transactions, a data image of the data obtained from a data operation may not be recorded in response to a delete operation or a read / query operation. The identified metadata and the recorded data image (if any) are then stored in a "structured audit log," which is associated with the record or row identifier of the record or row of the table, list, or index of the database on which such structured information was obtained. As used herein, a "structured audit log" refers to a data structure that stores the audit information (e.g., metadata, data images of records, etc.). In one embodiment, the structured audit log is stored in a storage device (e.g., memory, disk unit) of the database management system, wherein future queries for auditing the database can access such a stored structured audit log to obtain audit information about the database. In one embodiment, appropriate audit information is obtained from a stored structured audit log by matching the row or record identifier associated with the query with the record or row identifier associated with a structured audit log containing the requested database audit information. In this way, the accuracy and efficiency of database auditing are improved by utilizing structured audit logs.Furthermore, this approach represents an improvement in the technical field of database auditing.

[0183] The technical solutions provided in this disclosure cannot be implemented in a person's mind or by a person using pen and paper. That is, without the use of a computer, the technical solutions provided in this disclosure cannot be realized in a person's mind or by a person using pen and paper within any reasonable amount of time and under any reasonable expectation of accuracy.

[0184] In one embodiment of this disclosure, a computer-implemented method for improving the accuracy and efficiency of database auditing includes analyzing tables, lists, or indexes of a database to identify metadata, said metadata including time-series data, user data, Internet Protocol (IP) addresses, and operational data. The method further includes associating the identified metadata with corresponding records or rows of the analyzed tables, lists, or indexes of the database. The method also includes determining, based on corresponding data operations, whether to record a data image associated with a record or row of the analyzed table, list, or index of the database. Furthermore, the method includes storing the identified metadata and the recorded data image (if any) in a structured audit log, said structured audit log being associated with a record or row identifier of a record or row of the analyzed table, list, or index of the database.

[0185] Furthermore, in one embodiment of this disclosure, the method further includes recording an image of the data generated by the read operation when concurrent transactions exist. Additionally, the method includes, in response to storing the recorded image of the data generated by the read operation in a structured audit log when concurrent transactions exist, obtaining dirty read data relating to transaction concurrency from the structured audit log.

[0186] Furthermore, in one embodiment of this disclosure, the method further includes recording Structured Query Language (SCL) statements as data maps of Data Definition Language (DML) operations. The method also includes recording SCL expressions as data maps of Batch Data Manipulation Language (BDL) operations.

[0187] Furthermore, in one embodiment of this disclosure, the method further includes linking a query of a record or row in an analyzed table, list, or index of the database to a previously executed query of that record or row in the analyzed table, list, or index of the database.

[0188] Furthermore, in one embodiment of this disclosure, the method further includes storing pointers in the structured audit log to link queries on records or rows of analyzed tables, lists, or indexes of the database to previously executed queries on those records or rows of the analyzed tables, lists, or indexes of the database.

[0189] Furthermore, in one embodiment of this disclosure, the method further includes receiving a query request to audit the database. The method also includes identifying record or row identifiers of tables, lists, or indexes in the database in conjunction with the query request to audit the database.

[0190] Furthermore, in one embodiment of this disclosure, the method further includes retrieving audited information from the structured audit log in response to a match between an identified record or row identifier associated with a query request and a record or row identifier associated with a structured audit log.

[0191] Other embodiments of the above methods are systems and computer program products.

[0192] Various embodiments of this disclosure have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for improving the accuracy and efficiency of database auditing, the method comprising: Analyze the tables, lists, or indexes of the database to identify metadata, which includes time-series data, user data, Internet Protocol (IP) addresses, and operational data; Associate the identified metadata with the corresponding record or row in the table, list, or index of the database being analyzed; Based on the corresponding data operation, determine whether to record the data image associated with the record or row of the table, list, or index of the analysis in the database; as well as The identified metadata and data images of the records are stored in a structured audit log, which is associated with the record or row identifier of the record or row in the table, list, or index of the database being analyzed.

2. The method according to claim 1, further comprising: When concurrent transactions exist, record the image of the data generated by the read operations; as well as In response to the presence of the concurrent transaction, the record image of the data generated by the read operation is stored in the structured audit log, and dirty read data involving transaction concurrency is obtained from the structured audit log.

3. The method according to claim 1, further comprising: Record Structured Query Language (SCL) and Data Definition Language (DDL) statements as data maps of DDL operations; as well as Record Structured Query Language expressions as data maps for Batch Data Manipulation Language operations.

4. The method according to claim 1, further comprising: Link a query of a record or row in a table, list, or index of the analysis in the database to a previously executed query of the record or row in the table, list, or index of the analysis in the database.

5. The method of claim 4, further comprising: The structured audit log stores pointers to link the query of the record or row of the table, list, or index of the analysis in the database to a previously executed query of the record or row of the table, list, or index of the analysis in the database.

6. The method of claim 1, further comprising: Receive query requests to audit the database; as well as In conjunction with the query requests audited in the database, identify the record or row identifiers of the tables, lists, or indexes in the database.

7. The method of claim 6, further comprising: In response to a match between the identified record or row identifier associated with the query request and the record or row identifier associated with the structured audit log, audited information is retrieved from the structured audit log.

8. A computer program product for improving the accuracy and efficiency of database auditing, the computer program product comprising one or more computer-readable storage media having program code implemented therewith, the program code including program instructions for implementing the steps according to any one of claims 1 to 7.

9. A system comprising: A memory for storing computer programs used to improve the accuracy and efficiency of database auditing; as well as A processor connected to the memory, wherein the processor is configured to execute program instructions of the computer program to implement the steps according to any one of claims 1 to 7.

Images