Operation log management method, device, medium, equipment and program product

Abstract

This application provides a method, apparatus, computer-readable storage medium, electronic device, and computer program product for managing operation logs, relating to the field of data processing technology. The method includes: generating an operation transaction for a first database based on a database operation request; sending the operation transaction to a second server, whereby the second server is an instance server of the first database, to execute the operation transaction on the first database; if the operation transaction is successfully executed, generating a first message about the operation transaction; and adding the first message to a first message queue, so that a message consumer can retrieve first information from the first message queue and parse the first message to obtain the operation log corresponding to the operation transaction. This application provides a non-intrusive operation log management scheme for database operations, which can decouple database operation logic from operation log processing logic, thereby not increasing the burden on the execution of database operation logic.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for managing operation logs, an apparatus for managing operation logs, a computer-readable storage medium, an electronic device, and a computer program product. Background Technology

[0002] By managing the operation logs, users' needs for viewing operation logs can be met.

[0003] Related technologies offer embedded operation log management solutions, embedding logging functionality into business logic code. Specifically, operation log recording is executed within the same transaction as business logic operations, allowing logging code to be added at key points in the business logic (such as start, end, and exception handling) to capture the corresponding operation logs. These log records may include fields such as timestamps, operation types, user information, and operation results. Furthermore, the operation logs can be written to a database for user viewing.

[0004] However, the operation log management solutions provided by the relevant technologies, due to their tight coupling with business logic code, will increase the burden on the execution of business logic, thereby having a certain impact on the performance of the business system. Summary of the Invention

[0005] This application provides an operation log management method, operation log management device, computer-readable storage medium, electronic device, and computer program product. It provides a non-intrusive operation log management solution for database operations, which can decouple database operation logic from operation log processing logic, thereby not increasing the burden on the execution of database operation logic.

[0006] In a first aspect, embodiments of this application provide a method for managing operation logs, applied to a first server. The method includes: generating an operation transaction for a first database based on a database operation request; sending the operation transaction to a second server to execute the operation transaction on the first database through the second server, wherein the second server is an instance server of the first database; generating a first message for the operation transaction if the operation transaction is successfully executed; and adding the first message to a first message queue so that a message consumer can obtain the first message from the first message queue and parse the first message to obtain the operation log corresponding to the operation transaction.

[0007] In an exemplary embodiment, based on the above scheme, the method further includes: the message consumer writes the operation log into an operation log table, wherein the operation log table is stored in a second database.

[0008] In an exemplary embodiment, based on the above scheme, the method further includes: sending a log acquisition request to a third server to request the third server to send incremental operation logs from the second database since the last synchronization, wherein the third server is an instance server of the second database; receiving the incremental operation logs sent by the third server and parsing the incremental operation logs to extract the corresponding data change events; and sending the data change events to a fourth server so that the fourth server stores the data change events in the third database, wherein the fourth server is an instance server of the third database, and the third database is used to provide online analytical processing (OLAP) queries for the data change events.

[0009] In an exemplary embodiment, based on the above scheme, after extracting the corresponding data change event, the method further includes: adding the data change event to a second message queue so that the first server or the log query server can obtain the data change event from the second message queue; wherein, the log query server is used to receive a log query request and return the log query result corresponding to the log query request.

[0010] In an exemplary embodiment, based on the above scheme, before sending the data change event to the fourth server, the method further includes: for the i-th message, extracting key information from the i-th message, where i is a positive integer; performing a hash calculation based on the key information in the i-th message to obtain the i-th hash value corresponding to the i-th message; and, if it is determined that the j-th hash value corresponding to the j-th message is the same as the i-th hash value, if the data change event corresponding to the j-th message has already been sent to the fourth server, then cancel sending the i-th message to the fourth server; if the j-th message has already been parsed, then cancel parsing the i-th message; where j is a positive integer not equal to i.

[0011] In an exemplary embodiment, based on the above scheme, after sending the data change event to the fourth server, the method further includes: verifying, through a reconciliation mechanism, whether the data change event sent to the third database is consistent with the data change event stored in the second database; if there is an inconsistent target change event, then reprocessing the target change event through a replay mechanism.

[0012] In an exemplary embodiment, based on the above scheme, after extracting the corresponding data change event, the method further includes: persisting the data change event to a state table, so as to save the parsed data change event through the state table.

[0013] In an exemplary embodiment, based on the above scheme, the above operation transaction carries at least one of the following information: operator information, operation platform information, operation time, and tracking identifier, and different operation transactions correspond to different tracking identifiers; the above first message includes at least one of the following information: information of the operated object, field values ​​before and after the operation, operation time, operator information, operation platform information, and tracking identifier.

[0014] In an exemplary embodiment, based on the above scheme, the method further includes: receiving a log query request, the log query request including filtering conditions; determining a query statement according to the filtering conditions, wherein the query statement is used by a third database to perform a query operation to determine a target tracking identifier that meets the filtering conditions; inputting the query statement to a fourth server, so that the fourth server queries the third database according to the query statement and determines the log query result, wherein the fourth server is an instance server of the third database, the third database is used to provide online analytical processing (OLAP) queries for the data change events; and receiving the log query result.

[0015] In an exemplary embodiment, based on the above scheme, receiving the log query result includes: performing an optimization operation on the log query result, wherein the optimization operation includes at least one of the following: a classification operation and a translation operation.

[0016] Secondly, embodiments of this application provide an operation log management device configured on a first server. The device includes: an operation transaction generation module configured to generate an operation transaction for a first database based on a database operation request; an operation transaction sending module configured to send the operation transaction to a second server, so that the operation transaction can be executed on the first database through the second server, wherein the second server is an instance server of the first database; a message generation module configured to generate a first message about the operation transaction if the operation transaction is determined to be successfully executed; and a message sending module configured to add the first message to a first message queue, so that a message consumer can retrieve the first message from the first message queue and parse the first message to obtain the operation log corresponding to the operation transaction.

[0017] In an exemplary embodiment, based on the above solution, the apparatus further includes: a writing module;

[0018] The aforementioned writing module is used to write the aforementioned operation log into the operation log table through the aforementioned message consumer, wherein the aforementioned operation log table is stored in the second database.

[0019] In an exemplary embodiment, based on the above solution, the apparatus further includes: a synchronization module;

[0020] The synchronization module is configured to: send a log retrieval request to a third server, requesting the third server to send incremental operation logs from the second database since the last synchronization, wherein the third server is an instance server of the second database; receive the incremental operation logs sent by the third server and parse the incremental operation logs to extract the corresponding data change events; and send the data change events to a fourth server, so that the fourth server stores the data change events in the third database, wherein the fourth server is an instance server of the third database, and the third database is used to provide online analysis and processing (OLAP) queries for the data change events.

[0021] In an exemplary embodiment, based on the above scheme, the synchronization module is further configured to: after extracting the corresponding data change event, add the data change event to a second message queue so that the first server or the log query server can obtain the data change event from the second message queue; wherein, the log query server is configured to receive a log query request and return the log query result corresponding to the log query request.

[0022] In an exemplary embodiment, based on the above scheme, the synchronization module is further configured to: before sending the data change event to the fourth server, for the i-th message, extract key information from the i-th message, where i is a positive integer; perform hash calculation based on the key information in the i-th message to obtain the i-th hash value corresponding to the i-th message; and, if it is determined that the j-th hash value corresponding to the j-th message is the same as the i-th hash value, if the data change event corresponding to the j-th message has already been sent to the fourth server, then cancel sending the i-th message to the fourth server; if the j-th message has already been parsed, then cancel parsing the i-th message; where j is a positive integer not equal to i.

[0023] In an exemplary embodiment, based on the above scheme, the synchronization module is further configured to: after sending the data change event to the fourth server, verify through a reconciliation mechanism whether the data change event sent to the third database is consistent with the data change event stored in the second database; if there is an inconsistent target change event, reprocess the target change event through a replay mechanism.

[0024] In an exemplary embodiment, based on the above scheme, the synchronization module is further configured to: after extracting the corresponding data change event, persist the data change event to a status table, so as to save the parsed data change event through the status table.

[0025] In an exemplary embodiment, based on the above scheme, the above operation transaction carries at least one of the following information: operator information, operation platform information, operation time, and tracking identifier, and different operation transactions correspond to different tracking identifiers; the above first message includes at least one of the following information: information of the operated object, field values ​​before and after the operation, operation time, operator information, operation platform information, and tracking identifier.

[0026] In an exemplary embodiment, based on the above solution, the apparatus further includes: a query module;

[0027] The query module described above is used to: receive log query requests, which include filtering conditions;

[0028] The query statement is determined based on the above filtering conditions. This query statement is used by the third database to perform a query operation to determine the target tracking identifier that meets the above filtering conditions. The query statement is then input to the fourth server, so that the fourth server queries the third database based on the query statement and determines the log query result. The fourth server is an instance server of the third database, which is used to provide online analytical processing (OLAP) queries for the above data change events. The fourth server also receives the log query result.

[0029] In an exemplary embodiment, based on the above scheme, the query module is specifically used to: optimize the log query results, wherein the optimization operation includes at least one of the following: classification operation and translation operation.

[0030] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the operation log management method provided in the first aspect.

[0031] Fourthly, embodiments of this application provide a chip for implementing the operation log management method provided in the first or second aspect above. Specifically, the chip includes a processor for retrieving and running a computer program from a memory, causing a device equipped with the chip to execute the operation log management method provided in the first aspect above.

[0032] Fifthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to execute the operation log management method provided in the first aspect.

[0033] Sixthly, embodiments of this application provide a computer program product, including computer program instructions, which cause a computer to execute the operation log management method provided in the first aspect.

[0034] In a seventh aspect, embodiments of this application provide a computer program that, when run on a computer, causes the computer to execute the operation log management method provided in the first aspect.

[0035] In summary, the operation log management scheme provided in this application embodiment ensures the smooth execution of database operation logic. Specifically, it generates an operation transaction for the first database and sends the transaction to the instance server of the first database for execution. On the other hand, the first server, acting as a message producer, generates a first message about the operation transaction and adds it to a first message queue. This allows the message consumer to retrieve the first message from the queue and parse it to obtain the operation log corresponding to the operation transaction. Therefore, the scheme provided in this application embodiment decouples the operation log processing logic from the database business logic, ensuring that the first database business system is not interfered with by log recording operations during task execution, thus guaranteeing the normal operation and response speed of the business process. Furthermore, by implementing asynchronous processing of operation logs through a message queue, the processing burden of operation logs is transferred to the message consumer. This non-intrusive operation log processing scheme makes the system more flexible in deployment and maintenance, reducing modifications to existing business code. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A schematic diagram of the system architecture of an application environment for an operation log management scheme provided in this application embodiment;

[0038] Figure 2 A flowchart illustrating an operation log management method provided in an embodiment of this application;

[0039] Figure 3 This application provides a schematic diagram of information interaction within a decoupled operation log management framework, as illustrated in an embodiment of the present application.

[0040] Figure 4This is a schematic diagram illustrating the information interaction of a method for synchronizing operation logs to a log query system, provided in an embodiment of this application.

[0041] Figure 5 This is a schematic diagram illustrating the information interaction of a method for querying operation logs provided in an embodiment of this application.

[0042] Figure 6 This is a flowchart illustrating a method for obtaining and querying operation logs according to an embodiment of this application;

[0043] Figure 7 A schematic block diagram of an operation log management device provided in an embodiment of this application;

[0044] Figure 8 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0046] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein. In embodiments of this application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0047] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0048] An operation log is a mechanism for recording various operations and events that occur within a system. Specifically, it can record all change activities related to database interaction, such as inserting, updating, and deleting records. Operation logs have multiple applications. For example, they can be used for auditing and investigation, ensuring that it's possible to trace who performed which database operations and when, helping to maintain data integrity and security. For troubleshooting, they can help developers understand the system's behavior patterns and quickly locate the cause when problems occur. They can also be used for performance optimization; by analyzing frequent operation types and their impact, performance bottlenecks can be identified and improvement measures can be taken.

[0049] Related technologies also provide a solution for obtaining operation logs through database triggers. Specifically, triggers are set on database tables to capture data changes and record them in a log. These triggers automatically initiate log recording operations when data operations occur. However, setting up and managing triggers is relatively complex, and they can easily become performance bottlenecks in high-concurrency scenarios, making them unsuitable for real-time processing of large-scale log data.

[0050] Related technologies also provide a centralized log collection system, specifically using a centralized log collection system (such as the ELK (Elasticsearch, Logstash, Kibana) stack) to collect operational logs from various business systems into a central log management system. However, centralized log collection systems are complex, have high deployment and maintenance costs, poor real-time performance, and are not easy to achieve high availability.

[0051] Related technologies also offer solutions for obtaining operation logs based on log broker middleware. Specifically, standalone log broker middleware, such as Fluentd or Logstash, is used to collect application log data and forward it to a log storage system. However, solutions based on log broker middleware also increase system complexity and operational costs, and may experience latency and low real-time performance during data transmission.

[0052] To address the aforementioned issues, the solution provided in this application embodiment includes, on the one hand, a first server ensuring the smooth execution of database operation logic. Specifically, it generates an operation transaction for the first database and sends the operation transaction to the instance server of the first database to execute the operation transaction on the first database. On the other hand, the first server, acting as a message producer, generates a first message about the operation transaction and adds the first message to a first message queue, allowing the message consumer to retrieve the first message from the first message queue and parse it to obtain the operation log corresponding to the operation transaction. It is evident that the solution provided in this application embodiment decouples the operation log processing logic from the database business logic, ensuring that the first database business system is not interfered with by log recording operations during task execution, thus guaranteeing the normal operation and response speed of the business process. Simultaneously, by implementing asynchronous processing of operation logs through a message queue, the processing burden of operation logs is transferred to the message consumer. This non-intrusive operation log processing solution makes the system more flexible in deployment and maintenance, reduces modifications to existing business code, and is suitable for real-time processing of large-scale log data.

[0053] The following is through Figure 1 An exemplary system architecture of the embodiments of this application will be described.

[0054] For example, Figure 1 This is a schematic diagram of the system architecture 100 for an application environment of an operation log management scheme provided in an embodiment of this application. (See diagram below.) Figure 1 As shown, the system architecture of the implementation environment of this application embodiment may include: terminal 102, first server 104, and first database 106, etc.

[0055] For example, terminal 102 can be used to input information for database operations, such as providing an interface. For example, terminal 102 may include computers, smartphones, tablets, smart voice interaction devices, smart home appliances, in-vehicle terminals, aircraft, wearable smart devices, medical devices, etc., but is not limited to these. First server 104 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers. Database 106 can be a relational database such as MySQL (Structured Query Language).

[0056] In an exemplary embodiment, reference is made to Figure 1Terminal 102 obtains the user's database operation request and sends it to the first server 104 in the background. The first server 104 executes the database business processing logic. Specifically, the first server 104 generates an operation transaction for the first database based on the database operation request. For example, a transaction for inserting, updating, or deleting fields. The first server 104 sends the above operation transaction to the instance server of the first database (denoted as the second server, not shown in the figure) so that the second server can execute the operation transaction on the first database. That is, the second server implements operations such as inserting relevant fields in the first database. In the solution provided by this application embodiment, by decoupling the operation log processing logic from the database business logic, the first database business system will not be interfered with by log recording operations when performing tasks, ensuring the normal operation and response speed of the business process.

[0057] If the aforementioned operation transaction is successfully executed, the second server sends a success message to the first server 104, which then executes the operation log processing logic. Specifically, a first message about the operation transaction is generated and added to a first message queue. This asynchronous processing of the operation logs via the message queue shifts the processing burden to the message consumer. This non-intrusive operation log processing scheme makes the system more flexible in deployment and maintenance, reducing modifications to existing business code.

[0058] It should be noted that, Figure 1 The system architecture of the implementation environment of the embodiments of this application is illustrated, but the system construction of the implementation environment of the embodiments of this application is not limited to... Figure 1 As shown.

[0059] The following describes in detail the operation log management method of this application through some embodiments. These embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0060] Figure 2 This is a flowchart illustrating an operation log management method P200 provided in an embodiment of this application. The execution entity of method P200 is a server, such as... Figure 1 The first server in China is 104.

[0061] In step S210, an operation transaction for the first database is generated based on the database operation request.

[0062] For example, the data stored in the first database can be divided into three levels: account level, advertisement level, and creative level. Each level maintains its own type of data table. Users can perform operations such as inserting, updating, or deleting on the data tables corresponding to any level through database operation requests.

[0063] For example, the user accesses the terminal (such as...) Figure 1 Terminal 102) inputs a database operation request. For example, if a user inputs information to modify an advertisement on the front-end interface, the front-end system will transmit the fields the user wishes to modify, etc., as a database operation request to the back-end server. The request header of the aforementioned database operation request may contain at least one of the following information:

[0064] - Operator information;

[0065] -Operating platform;

[0066] - Modified time.

[0067] After receiving the aforementioned database operation request, the backend server 104 performs business verification based on the operator information contained in the request header to ensure operation security. If the verification passes, an operation transaction is generated based on the information in the request header. A transaction is a collection of atomic operations in the database; a transaction can contain multiple SQL statements, which either all execute successfully or all are rolled back. Transactions ensure data consistency and integrity. By incorporating one or more change operations into an operation transaction, it can be guaranteed that if any step fails, the entire operation can be rolled back, maintaining data consistency in the database. It should be noted that, for ease of operation log management, all change operations contained in each operation transaction are associated with the same TraceId.

[0068] In an exemplary embodiment, reference is made to Figure 3 In step S310, the user sends a database operation request through the front-end terminal 300.

[0069] For example, the first server 104 can generate operation transactions through the storage module 301. Step S311 is performed exemplary: the change records contained in the database operation request are incorporated into at least one operation transaction. Further, step S312 is performed, whereby the storage module 301 submits the operation transaction to the instance server of the first database, such as the second server 302. The storage module 301 is responsible for generating operation transactions and interacting with the database. Since the storage module 301 communicates directly with the database, it can understand which operations should be executed as a group of atomic operations. Therefore, the storage module 301 is suitable for processing and managing operation transactions to generate reasonable operation transactions. For example, if the database operation request contains operations 1, 2, and 3 on fields, and the storage module determines that operations 1-3 are closely related and can be executed as a group of atomic operations, then these operations can be incorporated into the same operation transaction A; or, if the storage module determines that operations 1-2 are related, and operations 3 have a weaker dependency on the other two operations, then operations 1 and 2 can be incorporated into the same operation transaction B, and operations 3 can be treated as another operation transaction C.

[0070] It should be noted that different operation transactions correspond to different identity identifiers, denoted as "TraceIdentifier" (TraceId). TraceId is a globally unique identity identifier, meaning that different operation transactions correspond to different TraceIds.

[0071] It is understood that the aforementioned storage module 301 may be part of the first backend server or an independent service or component, depending on the system architecture design, and this application embodiment does not limit it in this regard.

[0072] Continue to refer to Figure 2 In step S220, the operation transaction is sent to the second server so that the operation transaction is executed on the first database through the second server, where the second server is the instance server of the first database.

[0073] Exemplary Reference Figure 3 After receiving the operation transaction submitted by the storage module 301, the second server 302 executes step S313: executes the operation transaction to modify the first database. If the execution is successful, the second server executes step S314, returning a success message to the storage module 301.

[0074] In step S230, if it is determined that the operation transaction was successfully executed, a first message about the operation transaction is generated.

[0075] As previously described, this embodiment uses a message queue to decompose the "producer" and "consumer," transferring the operation log processing logic to the message consumer. Specifically, a message is an information unit transmitted in the message queue, and a message refers to a data packet encapsulating specific information, specifically information containing database changes. Therefore, after determining that the field change in the first database is successful, the backend server (specifically, the aforementioned storage module) generates a first message regarding the operation transaction.

[0076] refer to Figure 3 In step S321, the storage module 301 constructs a first message about the above-mentioned operation transaction.

[0077] For example, storage module 301 extracts relevant information for constructing the first message, such as the modified table name, fields, field values ​​(before and after the change), modification time, operator information, operating platform, and trace identifier TraceId. Further, it uses the collected information to construct a message body, ensuring that all information is correctly formatted and conforms to the message queue system's protocol requirements. The message body is serialized to convert it into a transmittable format (e.g., JSON or Protocol Buffers) so that it can be transmitted over the network. Therefore, the message body of the first message contains the following:

[0078] - Information about the object being operated on, including the table name, field names, etc. that have been changed;

[0079] - Field values ​​before and after the operation;

[0080] - Operation time;

[0081] - Operator information;

[0082] - Operating platform information;

[0083] - The trace identifier, i.e., TraceId mentioned above.

[0084] Further, in step S322, the storage module sends the first message to the message queue system 303 to add the first message to the message queue (denoted as the first message queue).

[0085] For example, the aforementioned message queue system can be Apache Pulsar. Apache Pulsar is an open-source distributed message processing platform capable of asynchronously transmitting information between different applications or services, specifically providing high-throughput, low-latency message delivery capabilities, and supporting multi-tenancy and geographical replication. As mentioned above, in this embodiment, Apache Pulsar can serve as a message middleware. It is understood that the aforementioned message queue system 303 is not limited to Apache Pulsar, but can also be other message queue systems; this embodiment does not limit its use.

[0086] For example, the storage module 301 sends the first message to a specified topic in the message queue system Pulsar for message consumers to subscribe to.

[0087] Continue to refer to Figure 2 In step S240, the first message is added to the first message queue so that the message consumer can retrieve the first message from the first message queue and parse the first message to obtain the operation log corresponding to the operation transaction.

[0088] In this embodiment, the backend system adds the first message to a first message queue, thereby eliminating the direct dependency between message producers (such as backend servers) and message consumers (such as asynchronous task processing systems). The backend server only needs to send messages to the first message queue, and the message consumer can retrieve messages from the first message queue for processing. Even if they do not run at the same time, it will not affect each other's work. Simultaneously, the message queue allows asynchronous communication between producers and consumers, reducing synchronization waiting time and improving system response speed and throughput. The message queue can also be used to achieve asynchronous transmission and processing of log data, ensuring high system availability. For example, in high-concurrency and large-data-volume scenarios, the system can expand its capacity and processing power by adding nodes, ensuring system stability and reliability. Furthermore, the use of message queues also provides traffic shaping functionality, further enhancing the system's resilience.

[0089] Exemplary Reference Figure 3 After receiving the first message, message queue system 303 executes step S323: temporarily saves the first message to the first message queue, in order to further execute step S324: distributes the message to message consumers 304 that have subscribed to the corresponding topic as needed. The first message queue can also ensure the reliability of message delivery; even if the message consumer is temporarily unavailable, message queue system 303 will retain the message until it can be processed correctly.

[0090] In an exemplary embodiment, after the message consumer 304 reads a message from the message queue system 303, it executes step S325: parsing the information in the message body.

[0091] For example, after receiving a message, the message consumer first needs to deserialize the message body, converting it from the transmission format (such as JSON or Protocol Buffers) back to the original data structure. Then, it parses out specific information based on the fields in the message, such as operation time, table name (including primary key), field values ​​before and after the operation, TraceId, operator information such as a universally unique identifier (UID), operation platform information, etc.

[0092] For example, after receiving a message, the message consumer parses the message body for each message and extracts key information (such as table name + primary key + operation time + TraceId) used for hash calculation (e.g., MD5 value). It then calculates the MD5 hash value based on the extracted key information. The message consumer queries its internal deduplication table (which can be a dedicated database table or other storage structure) to see if there are any identical MD5 values. Specifically, if no identical MD5 value exists, the message is considered not a duplicate and processing continues, while the MD5 value is saved for subsequent deduplication. If identical MD5 values ​​exist, the message is considered a duplicate and is either discarded or recorded for further analysis. Specifically, if the j-th hash value corresponding to the j-th message is the same as the i-th hash value, and if the data change event corresponding to the j-th message has already been sent to the fourth server, then sending the i-th message to the fourth server is canceled. If the j-th message has already been parsed, then parsing the i-th message is canceled; j is a positive integer not equal to i. This avoids redundant operations and improves management efficiency.

[0093] As can be seen, the parsed information may include at least one of the following:

[0094] - Operation time;

[0095] - The name of the table being operated on (including the primary key);

[0096] - Field values ​​before and after the operation;

[0097] -TraceId;

[0098] - Operator information, such as the operator's identification UID;

[0099] - Operating platform information;

[0100] - Hash values ​​are like MD5 values.

[0101] The information parsed by the 304 error at the message consumer end can be stored in any database, such as the first database mentioned above.

[0102] To avoid occupying additional storage fields in the business database (the aforementioned first database), ensure data isolation, and prevent impacting the read / write pressure on the business database, in this embodiment, the operation log table can be stored in a second database independent of the first database, thus achieving separation from the business data. This not only avoids adding extra storage fields to the business database but also reduces its read / write pressure. Through this data isolation method, the storage and management of operation logs become more flexible and efficient, without negatively impacting the performance of the business database, thereby improving the overall system reliability and maintainability.

[0103] In an exemplary embodiment, the message consumer 304 can persistently store the parsed information in the operation log table of the second database. Thus, the operation log table contains logs of all user operations on the first database, such as operation time, operation type (insert, update, delete), specific values ​​of the data before and after the change, and operator information.

[0104] In an exemplary embodiment, for the operation logs parsed and processed by the message consumer (304 error), this application provides a method to synchronize them to the log query system. Specifically, this can be implemented through a component or module of the backend server (denoted as the synchronization module or Panama module). For example, Figure 4 This is a schematic diagram illustrating the information interaction of a method for synchronizing operation logs to a log query system, provided in an embodiment of this application.

[0105] refer to Figure 4 In step S411, the synchronization module 401 sends a log retrieval request to the third server 305.

[0106] The third server is the instance server of the second database. The aforementioned log retrieval request is used to request the third server to send the incremental operation logs from the second database since the last synchronization, thereby ensuring that only incremental data is synchronized from the operation log table, rather than all data, avoiding redundant operations.

[0107] In step S412, the third server 305 retrieves the incremental data from the operation log table stored in the second database. Then, in step S413, it transmits the incremental operation log to the synchronization module 401.

[0108] For example, since each operation transaction (or each message) corresponds to a unique trace ID, the incremental operation log can be determined based on the trace ID. Specifically, the second database can record the trace IDs of operation logs that have been synchronized to the synchronization module 401. Upon receiving the log retrieval request, it can determine whether incremental operation logs exist or which operation logs belong to incremental operation logs based on the synchronized trace IDs. The incremental operation logs are then sent to the synchronization module 401. For example, after transmitting the incremental operation logs, the second database records the synchronized trace IDs again for accuracy in the next synchronization process. Alternatively, the synchronization module 401 can record which operation logs have been synchronized and include the synchronized operation IDs in the log retrieval request, allowing the second database to directly determine the incremental operation logs based on the information carried in the log retrieval request.

[0109] In step S414, the synchronization module 401 receives and parses the incremental operation log to extract the corresponding data change events.

[0110] The aforementioned data change events are used to describe information such as insertion, update, and deletion operations on certain fields. For example, it could be: at xx (time point) yy (user identity identifier), the s field is changed, the field value before the change is a1, and the field value after the change is a2.

[0111] For example, the operation log obtained by the synchronization module 401 from the second database may be a binary log (Binlog). Since the Binlog exists in the form of a bit stream, in this case, the synchronization module 401 not only needs to extract key information from the incremental operation log, but also needs to parse this bit stream into a data format that is easy for users to view, and obtain the corresponding data change events.

[0112] For example, the information parsed by the message consumer 304 (i.e., the operation logs stored in the aforementioned second database) can use an application-layer defined format. Therefore, the operation records in the second database can have their content and format adjusted according to specific business needs, without being restricted by the internal mechanisms of the source system, thus offering flexibility. The application-layer defined format can be an easily understood and processed data format, thereby reducing subsequent development and maintenance costs. In this case, the synchronization module 401 can directly extract information from the incremental operation logs to determine the corresponding data change events.

[0113] For example, the above-mentioned synchronization module 401 can also persist the parsed data change events to a status table, so as to save the parsed data change events through the status table and avoid repeated parsing or omission of parsing.

[0114] Continue to refer to Figure 4 In step S415, the synchronization module 401 synchronizes the data change events in batches to the third database for online analytical processing (OLAP) query of the data change events.

[0115] For example, in order to successfully synchronize data change events to the third database, the synchronization module 401 can check the MD5 value to avoid duplicate records. To improve synchronization efficiency and reduce the burden on the third database, batch processing technology can be used to write the operation time, the name of the table being operated on (including the primary key), the field values ​​before and after the operation, the TraceId, operator information such as UID, and operation platform information into the operation log table of the ClickHouse database.

[0116] For example, to ensure the reliability of data change events, if a target change event fails to be successfully synchronized to the third database, the synchronization module 401 can reprocess the target change event through a replay mechanism until it is successfully synchronized to the third database. Specifically, the synchronization module 401 can maintain a status table that records the parsed data change events. If an omission or failure is detected, the synchronization module 401 can restart processing from the last successfully synchronized data change event point according to the status table, ensuring that all changes are correctly synchronized.

[0117] For example, to ensure the reliability of data change events, after sending the data change event to the third database, the synchronization module 401 can also verify whether the data in the third database is consistent with that in the second database through a reconciliation mechanism. Specifically, the data between the two databases can be compared periodically based on a tracking identifier or other auxiliary identifier to find differences and locate the inconsistent target change event. Next, for the target change event, the replay mechanism described above can be used to reprocess the target change event until it is successfully synchronized to the third database.

[0118] For example, the aforementioned third database can employ the open-source columnar database management system ClickHouse. ClickHouse offers high performance and real-time data processing capabilities, making it suitable for scenarios involving large-scale data analysis and real-time data querying. For instance, when processing terabyte-level (TB) datasets, ClickHouse provides faster query response times and higher concurrency capabilities, making real-time querying and analysis of operation logs more efficient and meeting the real-time analysis needs in a big data environment. Therefore, the solution provided in this application embodiment can efficiently support online queries even with TB-level data.

[0119] Referring to Table 1, compared to other database management systems, the query speed of ClickHouse, a database management system that is more OLAP-friendly, can be improved by 90%. It can be seen that the log query system provided in this application embodiment has higher query speed and stability.

[0120] Table 1

[0121]

[0122] Continue to refer to Figure 4 In this embodiment of the application, the parsed data change event can also be added to a message queue (denoted as the second message queue) so that more systems can subscribe to it. Specifically, after executing step S414, steps S415'-S417' can also be executed.

[0123] In step S415', the synchronization module 401 sends the data change event to the message queue system 303 to add it to the message queue so that the first service or other servers can retrieve the data change event from the message queue.

[0124] As mentioned above, the message queue system 303 can be Apache Pulsar. It is understood that the message queue system 303 is not limited to Apache Pulsar, but can also be other message queue systems, as described in this embodiment.

[0125] In step S416', message queue system 303 temporarily stores the received data change event in the second message queue. And, in step S417', the fourth server 402, where the third database (such as ClickHouse) resides, retrieves the data change event from the second message queue.

[0126] As can be seen, the log query system can subscribe to the corresponding topics in the message queue system to obtain these data change events, thereby achieving real-time synchronization with the database.

[0127] In such Figure 4In the illustrated embodiment, incremental operation logs are synchronized to a third database (such as ClickHouse) in real time or periodically via synchronization mechanisms such as a synchronization module (Panama module). This ensures that ClickHouse contains operation logs for all transactions that performed operations on the first database. Furthermore, in the third database (such as ClickHouse), data change events are typically designed into table structures suitable for analytical queries based on business needs, potentially including optimization measures such as partitioned tables and sparse indexes to improve query performance.

[0128] In an exemplary embodiment, data change events from a third database (such as ClickHouse) can be achieved through, for example... Figure 3 and Figure 4 The corresponding embodiment may also be determined by any of the following schemes.

[0129] Option 1 utilizes a log broker middleware (such as Fluentd or Logstash). Specifically, this involves deploying a log broker middleware (such as Fluentd or Logstash) to collect operation logs from the business system. The log broker middleware then transmits the collected operation log data to a centralized log storage system (such as Elasticsearch or ClickHouse). Because log broker middleware supports various input and output plugins, this approach offers more flexible integration. It also allows for unified management and processing of various log data types, simplifying the operation log management process. Furthermore, it provides rich data processing and filtering capabilities, enhancing the usability of log data.

[0130] Option 2 utilizes database change data capture (CDC) tools, such as Debezium, to capture operational changes in the database. Furthermore, the captured change data is transmitted to a log storage system (such as a third-party database like ClickHouse) via a message queue such as Kafka. This approach eliminates the need to add logging code to the business logic, minimizing modifications to the existing system. Additionally, CDC tools can capture database changes in real time, ensuring timely log data. This method is suitable for smooth integration with existing systems, reducing implementation complexity.

[0131] Option three can utilize an event-driven architecture. Specifically, whenever a user action occurs, the front-end or back-end encapsulates the action as an event and publishes it to an event bus (such as Kafka or RabbitMQ). A separate log processing service subscribes to these events and asynchronously writes them to a log storage system (i.e., a third-party database such as Elasticsearch or ClickHouse). This event-driven architecture minimizes dependencies between components, improving system flexibility and maintainability. It also supports real-time data processing in large-scale distributed systems and is suitable for high-frequency operation logging. Furthermore, this approach allows for easy addition of new consumers or producers to adapt to future changes in requirements.

[0132] Next, we can... Figure 5 This document introduces an example of log querying based on a third-party database (such as ClickHouse). Figure 5 This is a schematic diagram illustrating the information interaction of a method for querying operation logs provided in an embodiment of this application.

[0133] refer to Figure 5 In step S511, the first server receives a log viewing request sent by the front-end terminal 300, which includes filtering conditions.

[0134] For example, if a user wants to view the update process of field b in the advertising-level data table within the time period from time xx to time yy, they can input the above filter conditions through the front-end terminal 300. The front-end terminal 300 will then generate a log query request based on the above filter conditions and the user's personal information.

[0135] In step S512, the first server determines a query statement based on the filtering conditions, wherein the query statement is used by the third database to perform a query operation to determine the target tracking identifier that meets the above filtering conditions. In step S513, the query statement is input to the instance server (fourth server 402) of the third database.

[0136] The aforementioned target tracking identifier is the tracking identifier of the operation transaction that meets the filtering conditions in the above request.

[0137] For example, after receiving the log query request, the first server can construct an SQL query statement based on the filtering conditions carried in the request, specifically for finding TraceIDs that meet the conditions from the operation log table. Furthermore, leveraging the efficient query capabilities of the third-party database (ClickHouse), all TraceIDs that meet the above filtering conditions can be quickly retrieved. For example, queries can be performed based on a range of timestamp fields, and filters can be applied to other conditions (such as operator UID, operating platform, etc.).

[0138] In step S514, the instance server of the third database executes a query operation based on the received query statement to obtain the target query identifier. In step S515, the log query result corresponding to the target query identifier is returned to the first server.

[0139] In an exemplary embodiment, if the log query results are extensive, it is not advisable to load all the data at once. Therefore, the instance server of the third database can adopt a paginated query approach. Only a portion of the records are retrieved at a time, thereby reducing memory usage and network transmission volume.

[0140] In an exemplary embodiment, for each page of data, the instance server of the third database can use the previously found target trace ID list as conditions to further query related table names, field changes, operator UIDs, operation times, and other detailed information to improve query efficiency through batch processing. To further improve query efficiency, ClickHouse's distributed query function, pre-aggregated tables, or materialized views can also be considered.

[0141] In step S516, after obtaining the log query results, the first server optimizes the original log query results. And in step S517, the optimized log query results are provided to the front-end terminal.

[0142] In an exemplary embodiment, the first server categorizes and organizes the log query results according to the operation log level selected by the user, ensuring that the data returned to the front end is arranged in hierarchical order.

[0143] In an exemplary embodiment, to improve the user experience, the first server can also translate database field names from English to Chinese, making the final interface displayed to the user more user-friendly. Specifically, this can be achieved through a predefined mapping table, or the translated content can be dynamically generated.

[0144] It is understood that the first server may perform other optimization processing on the log query results according to preset requirements or actual needs, and this application embodiment does not limit this.

[0145] In step S518, after the front-end terminal obtains the optimized log query results, it determines the rendered log query interface and displays the log query interface.

[0146] In an exemplary embodiment, the front-end receives log query results in JSON or other serialized formats returned by the first server via an application programming interface (API). This data is further parsed to extract necessary information, such as operation log entries and hierarchical relationships. Then, based on the parsed data, the front-end uses a JavaScript framework (such as React, Vue.js, etc.) or a template engine to dynamically generate Hypertext Markup Language (HTML) elements, displaying the operation logs visually to the user in the form of tables, charts, etc. For example, the front-end terminal can also provide a user-friendly interactive experience, such as allowing users to click to expand for more details and switch between different display modes (such as list view, tree view, etc.).

[0147] like Figure 5 The provided embodiment illustrates a process from the front-end initiating a request, the back-end server processing the query, to the front-end displaying the results. Specifically, it involves how to effectively retrieve large amounts of data from ClickHouse, and also how to organize and present this data in a reasonable manner to meet user query needs. This embodiment, through measures such as pagination, categorization, and field translation, ensures the accuracy and readability of query results, while also improving the response speed and user experience of the log query system.

[0148] The above text passed Figures 2 to 5 The overall management solution for operation logs has been introduced. The following section combines... Figure 6 This paper introduces a specific solution for managing operation logs.

[0149] Figure 6 This is a flowchart illustrating a method for obtaining and querying operation logs according to an embodiment of this application.

[0150] In step S61, the advertiser initiates the process of modifying the advertisement.

[0151] In step S62, the processing logic for modifying the database is executed.

[0152] The specific implementation methods of steps S61 and S62 can be referred to the embodiments corresponding to steps S310-S313, and will not be repeated here.

[0153] In step S63, it is determined whether the database modification was successful. If the modification was unsuccessful, the log operation-related procedures are not executed. If the modification was successful, step S64 is executed.

[0154] In step S64, a message is constructed and sent to the message queue system Pulsar.

[0155] The specific implementation of step S64 can be found in the embodiments corresponding to steps S321-S323, and will not be repeated here.

[0156] In step S65, the message consumer receives the Pulsar message.

[0157] In step S66, the message consumer writes the relevant information into the operation log table (MySQL).

[0158] The specific implementation of steps S65-S66 can be referred to the embodiments corresponding to steps S324-S326, and will not be repeated here.

[0159] In step S67, the Panama synchronization module synchronizes the operation log to ClickHouse.

[0160] The specific implementation of step S67 can be found in the following reference. Figure 4 The corresponding implementation examples will not be described in detail here.

[0161] In step S68, ClickHouse is queried to determine the advertiser's database modification logs.

[0162] The specific implementation of step S68 can be found in the following reference. Figure 5 The corresponding implementation examples will not be described in detail here.

[0163] This application provides an efficient, loosely coupled, and non-intrusive operation log management solution that meets users' needs for high real-time performance, high availability, and convenient maintenance of operation logs. Specifically, firstly, this application introduces a first message queue to asynchronously write operation logs to the operation log table, avoiding direct burden on the main database, improving overall system performance and response speed, and reducing the burden on the business system. Secondly, this application uses a non-intrusive method to record operation logs, separating the recording logic of the operation logs from the database business logic, reducing system coupling, facilitating maintenance and expansion, and thus improving system maintainability and scalability. Thirdly, this application utilizes a first message queue for asynchronous transmission and synchronization of operation logs, ensuring that log data can be delivered to a third database (such as ClickHouse) in real time, thereby achieving efficient real-time querying. Fourthly, through distributed architecture design and asynchronous processing mechanisms, this application ensures high availability even under high concurrency and large data volume conditions, preventing the stable operation of the main business system from being affected by the recording of operation logs. Fifthly, the architecture provided in this application simplifies the overall system design, reduces reliance on centralized log collection systems and log broker middleware, thereby reducing operation and maintenance costs.

[0164] The above text combined Figures 1 to 6 This paper describes an embodiment of the operation log management method of this application, which is illustrated below in conjunction with... Figure 7 This application describes an embodiment of the operation log management device.

[0165] Figure 7 This is a schematic block diagram of an operation log management device 700 provided in an embodiment of this application.

[0166] refer to Figure 7 The operation log management device 700 provided in this application embodiment is configured on the first server, such as... Figure 1 The first server 104 is shown in the figure. The device includes: an operation transaction generation module 710, an operation transaction sending module 720, a message generation module 730, and a message sending module 740.

[0167] The operation transaction generation module 710 is used to generate an operation transaction for the first database according to a database operation request; the operation transaction sending module 720 is used to send the operation transaction to a second server so that the operation transaction can be executed on the first database through the second server, wherein the second server is an instance server of the first database; the message generation module 730 is used to generate a first message about the operation transaction if it is determined that the operation transaction is executed successfully; and the message sending module 740 is used to add the first message to a first message queue so that the message consumer can obtain the first message from the first message queue and parse the first message to obtain the operation log corresponding to the operation transaction.

[0168] In an exemplary embodiment, based on the above solution, the apparatus further includes: a writing module;

[0169] The aforementioned writing module is used to write the aforementioned operation log into the operation log table through the aforementioned message consumer, wherein the aforementioned operation log table is stored in the second database.

[0170] In an exemplary embodiment, based on the above solution, the apparatus further includes: a synchronization module;

[0171] The synchronization module is configured to: send a log retrieval request to a third server, requesting the third server to send incremental operation logs from the second database since the last synchronization, wherein the third server is an instance server of the second database; receive the incremental operation logs sent by the third server and parse the incremental operation logs to extract the corresponding data change events; and send the data change events to a fourth server, so that the fourth server stores the data change events in the third database, wherein the fourth server is an instance server of the third database, and the third database is used to provide online analysis and processing (OLAP) queries for the data change events.

[0172] In an exemplary embodiment, based on the above scheme, the synchronization module is further configured to: after extracting the corresponding data change event, add the data change event to a second message queue so that the first server or the log query server can obtain the data change event from the second message queue; wherein, the log query server is configured to receive a log query request and return the log query result corresponding to the log query request.

[0173] In an exemplary embodiment, based on the above scheme, the synchronization module is further configured to: before sending the data change event to the fourth server, for the i-th message, extract key information from the i-th message, where i is a positive integer; perform hash calculation based on the key information in the i-th message to obtain the i-th hash value corresponding to the i-th message; and, if it is determined that the j-th hash value corresponding to the j-th message is the same as the i-th hash value, if the data change event corresponding to the j-th message has already been sent to the fourth server, then cancel sending the i-th message to the fourth server; if the j-th message has already been parsed, then cancel parsing the i-th message; where j is a positive integer not equal to i.

[0174] In an exemplary embodiment, based on the above scheme, the synchronization module is further configured to: after sending the data change event to the fourth server, verify through a reconciliation mechanism whether the data change event sent to the third database is consistent with the data change event stored in the second database; if there is an inconsistent target change event, reprocess the target change event through a replay mechanism.

[0175] In an exemplary embodiment, based on the above scheme, the synchronization module is further configured to: after extracting the corresponding data change event, persist the data change event to a status table, so as to save the parsed data change event through the status table.

[0176] In an exemplary embodiment, based on the above scheme, the above operation transaction carries at least one of the following information: operator information, operation platform information, operation time, and tracking identifier; the above first message includes at least one of the following information: information of the operated object, field values ​​before and after the operation, operation time, operator information, operation platform information, and tracking identifier.

[0177] In an exemplary embodiment, based on the above solution, the apparatus further includes: a query module;

[0178] The query module described above is used to: receive log query requests, which include filtering conditions;

[0179] The query statement is determined based on the above filtering conditions. This query statement is used by the third database to perform a query operation to determine the target tracking identifier that meets the above filtering conditions. The query statement is then input to the fourth server, so that the fourth server queries the third database based on the query statement and determines the log query result. The fourth server is an instance server of the third database, which is used to provide online analytical processing (OLAP) queries for the above data change events. The fourth server also receives the log query result.

[0180] In an exemplary embodiment, based on the above scheme, the query module is specifically used to: optimize the log query results, wherein the optimization operation includes at least one of the following: classification operation and translation operation.

[0181] The operation log management device provided in this application embodiment is configured on a first server. On one hand, the operation transaction generation module and operation transaction sending module in the first server can ensure the smooth execution of database operation logic. Specifically, the operation transaction generation module generates an operation transaction about the first database, and then the operation transaction sending module sends the operation transaction to the instance server of the first database to execute the operation transaction on the first database. On the other hand, the message generation module in the first server, as a message producer, generates a first message about the operation transaction, and further, the message sending module adds the first message to a first message queue so that the message consumer can obtain the first message from the first message queue and parse the first message to obtain the operation log corresponding to the operation transaction. It can be seen that the solution provided in this application embodiment decouples the operation log processing logic from the database business logic, so that the first database business system will not be disturbed by log recording operations when executing tasks, ensuring the normal operation and response speed of the business process. At the same time, the asynchronous processing of operation logs is realized through message queues, transferring the processing burden of operation logs to the message consumer. This non-intrusive operation log processing solution makes the system more flexible in deployment and maintenance, reducing the need for modifications to existing business code.

[0182] Furthermore, the write module in the first server stores the operation logs in a separate database to ensure data isolation and security. The synchronization module of the first server synchronizes the operation logs to the third database, ClickHouse, which is well-suited for OLAP query scenarios and can efficiently handle large-scale data queries. The query module of the first server, by querying the ClickHouse database, can achieve efficient and real-time display of the operation logs.

[0183] It should be understood that, as Figure 7The embodiment of the operation log management device shown corresponds to the embodiment of the operation log management method described above, and a similar description can be found in the method embodiment. To avoid repetition, it will not be repeated here. Specifically, through... Figure 7 The information interaction between the various modules in the operation log management device shown can execute the above-described embodiment of the operation log management method, through, as... Figure 8 The information interaction between the various modules in the operation log management device shown can execute the above-described operation log management method embodiments. For the sake of brevity, the aforementioned and other operation and / or function corresponding method embodiments of each module in the device will not be described again here.

[0184] The above description, in conjunction with the accompanying drawings, describes the operation and maintenance related apparatus of the software agent according to the embodiments of this application from the perspective of functional modules. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by the integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0185] This application also provides an electronic device.

[0186] Figure 8 This is a schematic block diagram of an electronic device 800 provided in an embodiment of this application. As described above, the operation and maintenance related devices of the software agent can be deployed in, for example... Figure 8 The electronic device shown can therefore be used to perform the above-described operation log management method.

[0187] like Figure 8 As shown, the electronic device 800 may include:

[0188] The system includes a memory 810 and a processor 820. The memory 810 stores a computer program 830 and transfers the program code 830 to the processor 820. In other words, the processor 820 can retrieve and run the computer program 830 from the memory 810 to implement the methods described in the embodiments of this application.

[0189] For example, the processor 820 can be used to execute the steps in the above-described operation log management method according to the instructions in the computer program 830, or to execute the steps in the above-described operation log management method.

[0190] In some embodiments of this application, the processor 820 may include, but is not limited to:

[0191] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0192] In some embodiments of this application, the memory 810 includes, but is not limited to:

[0193] Volatile memory and / or non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM).

[0194] In some embodiments of this application, the computer program 830 may be divided into one or more modules, which are stored in the memory 810 and executed by the processor 820 to complete the operation log management method provided in this application, or to complete the steps in the operation log management method described above. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 830 in the electronic device.

[0195] like Figure 8 As shown, the electronic device 800 may further include:

[0196] Transceiver 840, which can be connected to processor 820 or memory 810.

[0197] The processor 820 can control the transceiver 840 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 840 may include a transmitter and a receiver. The transceiver 840 may further include antennas, and the number of antennas may be one or more.

[0198] It should be understood that the various components in the electronic device 800 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0199] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0200] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above-described method embodiments.

[0201] In other words, when implemented using software, it can be implemented wholly or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0202] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0204] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0205] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for managing operation logs, characterized in that, Applied to a first server, the method includes: Based on the database operation request, generate an operation transaction for the first database; The operation transaction is sent to the second server so that the operation transaction can be executed on the first database through the second server, where the second server is the instance server of the first database; If the operation transaction is determined to be successfully executed, a first message about the operation transaction is generated. The first message is added to the first message queue so that the message consumer can retrieve the first message from the first message queue and parse the first message to obtain the operation log corresponding to the operation transaction.

2. The method according to claim 1, characterized in that, The method further includes: The message consumer writes the operation log into an operation log table, which is stored in a second database.

3. The method according to claim 2, characterized in that, The method further includes: Send a log retrieval request to a third server to request the third server to send the incremental operation logs in the second database since the last synchronization, wherein the third server is the instance server of the second database; Receive the incremental operation log sent by the third server, and parse the incremental operation log to extract the corresponding data change events; The data change event is sent to a fourth server, which then stores the data change event in a third database. The fourth server is an instance server of the third database, which is used to provide online analytical processing (OLAP) queries for the data change event.

4. The method according to claim 3, characterized in that, After extracting the corresponding data change event, the method further includes: The data change event is added to the second message queue so that the first server or the log query server can retrieve the data change event from the second message queue; The log query server is used to receive log query requests and return the log query results corresponding to the log query requests.

5. The method according to claim 3, characterized in that, Before sending the data change event to the fourth server, the method further includes: For the i-th message, extract the key information from the i-th message, where i is a positive integer; The key information in the i-th message is used to perform a hash calculation to obtain the i-th hash value corresponding to the i-th message; If the hash value corresponding to the j-th message is the same as the i-th hash value, and if the data change event corresponding to the j-th message has been sent to the fourth server, then the sending of the i-th message to the fourth server is cancelled; if the j-th message has been parsed, then the parsing of the i-th message is cancelled; j takes the value of a positive integer not equal to i.

6. The method according to claim 3, characterized in that, After sending the data change event to the fourth server, the method further includes: The reconciliation mechanism verifies whether the data change events sent to the third database are consistent with the data change events stored in the second database. If inconsistent target change events exist, the target change events are reprocessed through a replay mechanism.

7. The method according to any one of claims 3 to 6, characterized in that, After extracting the corresponding data change event, the method further includes: The data change events are persisted to a state table to store the parsed data change events.

8. The method according to any one of claims 1 to 6, characterized in that, The operation transaction carries at least one of the following information: operator information, operation platform information, operation time, and tracking identifier. Different operation transactions correspond to different tracking identifiers. The first message includes at least one of the following: information about the object being operated on, field values ​​before and after the operation, operation time, operator information, operation platform information, and tracking identifier.

9. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Receive a log query request, the log query request including filtering conditions; The query statement is determined based on the filtering conditions, wherein the query statement is used by the third database to perform a query operation to determine the target tracking identifier that meets the above filtering conditions; The query statement is input to the fourth server, so that the fourth server queries the third database according to the query statement and determines the log query result. The fourth server is an instance server of the third database, which is used to provide online analysis and processing (OLAP) queries for the data change events. Receive the log query results.

10. The method according to claim 9, characterized in that, Receiving the log query result includes: The log query results are optimized, wherein the optimization operation includes at least one of the following: classification operation and translation operation.

11. A management device for operation logs, characterized in that, Configured on a first server, the device includes: The operation transaction generation module is used to generate operation transactions for the first database based on database operation requests. An operation transaction sending module is used to send the operation transaction to a second server so that the operation transaction can be executed on the first database through the second server, wherein the second server is an instance server of the first database; The message generation module is used to generate a first message about the operation transaction if it is determined that the operation transaction was successfully executed. The message sending module is used to add the first message to the first message queue so that the message consumer can retrieve the first message from the first message queue and parse the first message to obtain the operation log corresponding to the operation transaction.

12. A computer-readable storage medium, characterized in that, Used to store computer programs; The computer program causes the computer to execute the operation log management method as described in any one of claims 1 to 10.

13. An electronic device, wherein, Including processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the operation log management method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the operation log management method as described in any one of claims 1 to 10.

Images