Log aggregation method, device, computer equipment and storage medium

Abstract

The present application relates to a log aggregation method, device, computer equipment, storage medium and computer program product. The embodiments of the present application can be applied to the field of maps. The embodiments of the present application can also be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The method first obtains a binary log; extracts row format events from the binary log; compares the row format events with the stored events in the preset event buffer for event meta information; when the comparison result is that the event meta information is consistent, the row format events are placed in the preset event buffer, and when the comparison result is that the event meta information is inconsistent, the stored events in the preset event buffer are aggregated. After obtaining the binary log, the method of the present application aggregates the row format events with consistent binary log meta information through meta information comparison, thereby reducing the number of full table scans in the subsequent processing process, thereby improving the binary log communication efficiency between the master and slave machines.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log aggregation method, apparatus, computer device and storage medium. Background Art

[0002] With the development of computer technology and database technology, cloud technology has emerged. Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to achieve data computing, storage, processing and sharing. It is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application. It can form a resource pool and use it on demand, which is flexible and convenient. Cloud computing technology will become an important support.

[0003] Currently, in the cloud environment, the binary log communication between the master's binary log Dump thread and the slave's IO (Input / Output) thread is very fast and has very low latency. Therefore, the online master-slave delay is mainly caused by the slave's SQL (Structured Query Language) thread's slow application of binary logs. The update / delete of the table without a primary key is the main cause of the master-slave delay. This is mainly because the master generally only needs to perform a full table scan once to execute an update / delete statement, but it will generate multiple row-based logs. On the slave, each application of a row-based log will go through a full table scan (table scan). When a large transaction occurs and the amount of data in the table is very large, the cost of a full table scan is very high. If there are a lot of update or delete logs, multiple full table scan operations are required, which affects the efficiency of binary log communication between the master and slave. Summary of the invention

[0004] Based on this, it is necessary to provide a log aggregation method, device, computer equipment, computer-readable storage medium and computer program product that can be used to improve the efficiency of binary log communication between master and slave machines in response to the above technical problems.

[0005] In a first aspect, the present application provides a log aggregation method. The method comprises:

[0006] Get binary logs;

[0007] Extracting row format events in the binary log;

[0008] Comparing the row format event with the events stored in the preset event buffer for event meta information;

[0009] When the comparison result shows that the event meta information is consistent, the row format event is placed in the preset event buffer area; when the comparison result shows that the event meta information is inconsistent, the stored events in the preset event buffer area are aggregated.

[0010] In one embodiment, extracting the row format events in the binary log includes:

[0011] Reading events from the binary log in units of events;

[0012] Determine the event type of the event being read;

[0013] When the event type of the read event is a row format event type, the read event is extracted from the binary log, and the read event is used as a row format event.

[0014] In one embodiment, obtaining the binary log includes:

[0015] Get transaction request;

[0016] Determining a transaction type of the transaction processing request;

[0017] When the transaction type of the transaction processing request is a transaction without a primary key table, a binary log corresponding to the transaction processing request is obtained.

[0018] In one embodiment, when the comparison result is that the event meta information is inconsistent, aggregating the stored events in the preset event buffer area includes:

[0019] When the comparison result shows that the event meta information is inconsistent, the row format data of the event stored in the preset event buffer is obtained;

[0020] splicing the row format data of the stored events to obtain spliced ​​data;

[0021] Acquire an aggregation event according to the spliced ​​data;

[0022] The aggregated event is moved out of the preset event buffer area, and the preset event buffer area is cleared.

[0023] In one embodiment, obtaining an aggregate event according to the spliced ​​data includes:

[0024] Obtaining the initial stored events in the preset event buffer;

[0025] The row format data of the initial stored event is replaced with the spliced ​​data to obtain an aggregated event.

[0026] In one embodiment, after the aggregate event is moved out of the preset event buffer and the preset event buffer is cleared, the method further includes:

[0027] The aggregate event is executed, and the row format event is placed into the preset event buffer.

[0028] In a second aspect, the present application also provides a log aggregation device. The device includes:

[0029] Data acquisition module, used to obtain binary logs;

[0030] An event extraction module, used to extract row format events from the binary log;

[0031] An event comparison module, used for comparing event meta information of the row format event with events stored in a preset event buffer;

[0032] The log aggregation module is used to place the row format event into the preset event buffer when the comparison result shows that the event meta information is consistent, and to aggregate the stored events in the preset event buffer when the comparison result shows that the event meta information is inconsistent.

[0033] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0034] Get binary logs;

[0035] Extracting row format events in the binary log;

[0036] Comparing the row format event with the events stored in the preset event buffer for event meta information;

[0037] When the comparison result shows that the event meta information is consistent, the row format event is placed in the preset event buffer area; when the comparison result shows that the event meta information is inconsistent, the stored events in the preset event buffer area are aggregated.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0039] Get binary logs;

[0040] Extracting row format events in the binary log;

[0041] Comparing the row format event with the events stored in the preset event buffer for event meta information;

[0042] When the comparison result shows that the event meta information is consistent, the row format event is placed in the preset event buffer area; when the comparison result shows that the event meta information is inconsistent, the stored events in the preset event buffer area are aggregated.

[0043] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0044] Get binary logs;

[0045] Extracting row format events in the binary log;

[0046] Comparing the row format event with the events stored in the preset event buffer for event meta information;

[0047] When the comparison result shows that the event meta information is consistent, the row format event is placed in the preset event buffer area; when the comparison result shows that the event meta information is inconsistent, the stored events in the preset event buffer area are aggregated.

[0048] The log aggregation method, device, computer equipment, storage medium and computer program product are as follows: first, obtain a binary log; extract row format events from the binary log; compare the row format events with the stored events in the preset event buffer for event meta information; when the comparison result is that the event meta information is consistent, the row format events are placed in the preset event buffer; when the comparison result is that the event meta information is inconsistent, the stored events in the preset event buffer are aggregated. After obtaining the binary log, the method of the present application aggregates the row format events with consistent binary log meta information through meta information comparison, thereby reducing the number of full table scans in the subsequent processing process, thereby improving the binary log communication efficiency between the master and slave machines. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 An application environment diagram of a log aggregation method in an embodiment;

[0050] Figure 2 A flow chart of a log aggregation method in one embodiment;

[0051] Figure 3 A schematic diagram of a flow chart of steps for extracting row format events from a binary log in one embodiment;

[0052] Figure 4 A flowchart of the steps of obtaining binary logs in one embodiment;

[0053] Figure 5A flowchart of log aggregation steps by splicing data in a row format according to an embodiment;

[0054] Figure 6 A schematic diagram of a process for obtaining aggregated events in one embodiment;

[0055] Figure 7 A schematic diagram of a data structure for obtaining an aggregate event step in an embodiment;

[0056] Figure 8 A flow chart of a log aggregation method in another embodiment;

[0057] Fig. 9 is a structural block diagram of a log aggregation device in an embodiment;

[0058] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] The present application mainly relates to database technology in cloud technology. In short, a database can be regarded as an electronic file cabinet - a place to store electronic files. Users can add, query, update, delete, and other operations on the data in the file. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program. A database management system (DBMS) is a computer software system designed for managing databases, generally with basic functions such as storage, interception, security, and backup. Database management systems can be classified according to the database models they support, such as relational, XML (Extensible Markup Language); or according to the types of computers they support, such as server clusters and mobile phones; or according to the query language used, such as SQL (Structured Query Language), XQuery; or according to performance impulse focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages ​​at the same time.

[0061] In this article, it is important to understand that the terms involved are:

[0062] Binary log (binlog): It is the most important log in MySQL (relational database management system). It records all DDL (Data Definition Language) and DML (Data Manipulation Language) statements (except data query statements such as select and show), in units of events. It also includes the time consumed by the execution of the statement. MySQL's binary log is transaction-safe.

[0063] Master and slave: The master is the node that generates binary logs, and the slave is the node that consumes binary logs.

[0064] Binary log format: Indicates the format in which DML statements are stored in the binary log. In MySQL, there are three log formats: row-based log, statement-based log, and mixed-base log.

[0065] Row-based log: The log records the form in which each row of data is modified, and then the same data is modified on the slave side. In row mode, the binary log does not need to record the context-related information of the executed SQL statement, but only needs to record which record is modified and how it is modified. Therefore, the row log content will clearly record the details of each row of data modification, which is very easy to understand. In addition, there will be no problem that the stored procedures or functions in certain situations, as well as the calling and triggering of triggers, cannot be correctly replicated.

[0066] Event: It is a storage unit in the binary log, where the upper limit of the size of an event is fixed.

[0067] Row event: An event type that generates row-based logs from DML operations.

[0068] Row data: Row-based log data generated by DML operations and stored in Rowevent.

[0069] Full table scan: In the absence of any index, MySQL uses a full table scan to find a specific record.

[0070] Hash scan: In the absence of a primary key or unique index, the slave can use the records to be applied to hash, and then perform a full table scan to quickly match the logs to be applied, thereby speeding up the application of logs.

[0071] The log aggregation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the slave terminal 102 communicates with the host server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the host server 104, or it can be placed on the cloud or other servers. When the slave terminal 102 synchronizes the binary log with the host server 104 through the network, it is necessary to apply the binary log on the SQL thread of the slave terminal 102, and the speed of the binary log is too slow, which will cause delays between the online master and slave machines. Therefore, the log aggregation method of the present application can be used to improve the log application efficiency of the slave terminal 102, thereby reducing the delay between the master and slave machines and improving the communication efficiency. When performing log communication, the slave terminal 102 first obtains the binary log from the host server 104; extracts the row format event in the binary log; compares the row format event with the stored event in the preset event buffer area for event meta information; when the comparison result is that the event meta information is consistent, the row format event is placed in the preset event buffer area, and when the comparison result is that the event meta information is inconsistent, the stored events in the preset event buffer area are aggregated. Then, the host server 104 is communicated with according to the aggregated binary log data. Among them, the host server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The slave terminal 102 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. The slave terminal 102 and the server 104 can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0072] In one embodiment, Figure 2 As shown, a log aggregation method is provided, which is applied to Figure 1 Taking the slave terminal 102 in the example as an example, the method includes the following steps:

[0073] Step 201, obtaining binary logs.

[0074] The binary log is the most important log of MySQL, which records all DDL and DML statements in units of events and also includes the time consumed by the statement execution. The host server 104 generates the binary log, while the slave terminal 102 consumes the binary log.

[0075] Specifically, the log aggregation method of the present application is mainly applied to the slave terminal 102. Since the binary log communication delay between the binary log Dump thread of the host and the IO thread of the slave is mainly caused by the slow speed of the slave terminal 102 applying the log, the logs received by the slave terminal 102 can be aggregated through the present application to speed up the speed of the slave terminal 102 applying the log, thereby reducing the log communication delay between the master and the slave. First, when performing log aggregation, the slave terminal 102 can obtain the binary log sent by the host server 104 through the network, thereby aggregating the binary log.

[0076] Step 203: extract row format events from the binary log.

[0077] Among them, events are storage units in binary logs, and the upper limit of the size of events is fixed. Row format events are a type of event format. There are three formats for storing DML statements in binary logs, namely row-based logs, statement-based logs, and mixed row and statement formats. In row-based logs, each row of data is recorded as modified, and then the same data is modified on the slave side. In row mode, the binary log does not need to record the context-related information of the executed SQL statement, but only needs to record which record has been modified and how it has been modified. Therefore, the row log content will clearly record the details of each row of data modification, which is very easy to understand. In addition, there will be no problem that the stored procedures or functions in certain specific situations, as well as the calling and triggering of triggers cannot be correctly replicated.

[0078] Specifically, the log aggregation method of the present application is mainly used for row format events in binary logs, thereby aggregating binary logs, and binary logs contain three types of logs: row format-based logs, statement-based logs, and mixed row format and statement format logs. Therefore, after obtaining the binary logs, it is necessary to identify which events are event types of row format-based logs from the binary logs, so as to aggregate these events.

[0079] Step 205 : compare the event metadata of the row format event with the events stored in the preset event buffer.

[0080] The preset event buffer is an additional storage area for temporarily storing the extracted events. Event meta information refers to information used to describe event attributes, including the type of event, the table operated by the event, and the ID of the server where the event is generated.

[0081] Specifically, the log aggregation object of the present application is the row format event in the log. When performing aggregation, it is first necessary to determine whether two row format events can be aggregated into one event, and the basis for the judgment is the event meta information between different events. Therefore, after extracting a row format event, the event meta information can be compared with the stored events in the preset event buffer to determine whether the two can be aggregated. The scheme of the present application is to strictly judge all the event meta information in the two events before aggregation, but in fact some event meta information has no effect on aggregation. Therefore, in a specific embodiment, this part can be further optimized: some signs that have no effect on data consistency can be ignored during the comparison process, for example: there is a sign about the end of the statement (STMT_END_F). If this sign is ignored during the comparison process, then the logs generated by two similar statements on the host can be aggregated.

[0082] Step 207, when the comparison result shows that the event meta information is consistent, the row format event is placed in a preset event buffer area; when the comparison result shows that the event meta information is inconsistent, the stored events in the preset event buffer area are aggregated.

[0083] Specifically, event meta information consistency means that the row format event is completely consistent with the event meta information of the stored event in the preset event buffer. At this time, two different row format events can be aggregated into one event. If the comparison result is that the event meta information is inconsistent, then the data comparison ends and all N events stored in the preset event buffer are aggregated into one event, so that hash scanning can be used directly to execute the aggregated events in the subsequent process. In this way, only one full disk scan is needed to complete the execution of N events, while before, this required N full disk scans. This can effectively improve the efficiency of binary log communication between the master and slave machines.

[0084] The above log aggregation method first obtains the binary log; extracts the row format events in the binary log; compares the row format events with the stored events in the preset event buffer area for event meta information; when the comparison result is that the event meta information is consistent, the row format events are placed in the preset event buffer area, and when the comparison result is that the event meta information is inconsistent, the stored events in the preset event buffer area are aggregated. After obtaining the binary log, the method of the present application aggregates the row format events with consistent binary log meta information through event meta information comparison, thereby reducing the number of full table scans in the subsequent processing process, thereby improving the binary log communication efficiency between the master and slave machines.

[0085] In one embodiment, Figure 3 As shown, step 203 includes:

[0086] Step 302: Read events from the binary log in units of events.

[0087] Step 304: determine the event type of the event being read.

[0088] Step 306: When the event type of the read event is a row format event type, extract the read event from the binary log and use the read event as a row format event.

[0089] Among them, reading events from binary logs in units of events means reading events in the binary logs in sequence, taking one event each time for processing and comparison. Event types include row format event types, statement format event types, and mixed row format and statement format event types, which correspond to the three binary log formats respectively. After reading an event from the binary log, the event type corresponding to the read event can be determined, so as to determine whether it can be aggregated.

[0090] Specifically, when performing log aggregation processing, the solution of the present application first needs to read events one by one from the binary log, aggregate them in units of events, and only events of row format event types can be aggregated. Therefore, after reading an event, the event type of the read event can be determined first; only when it is determined that the event type of the read event is a row format event type, the read event is extracted from the binary log, and the read event is used as a row format event, otherwise continue to read the next event. Until an event of the row format event type is read, the read event is extracted from the binary log, thereby obtaining a row format event, and subsequent log aggregation is performed. In this embodiment, by judging the event type, row format events can be effectively extracted from the binary log to ensure the smooth progress of the log aggregation processing process.

[0091] In one embodiment, Figure 4As shown, step 201 includes:

[0092] Step 401, obtaining a transaction processing request.

[0093] Step 403: Determine the transaction type of the transaction processing request.

[0094] Step 405: When the transaction type of the transaction processing request is a transaction without a primary key table, a binary log corresponding to the transaction processing request is obtained.

[0095] In the database, a SQL statement can be called a basic operation. A transaction is a transaction that "packages" several SQL statements together to perform a complete task. A transaction consists of one or more basic operations, or one or more SQL statements. The most notable feature of a transaction is that all the SQL statements it contains are submitted to the database as a whole. Only when all the SQL statements are executed, the entire transaction is considered successful. Once a SQL statement fails to execute, the entire transaction fails. After a transaction fails, all SQL statements need to be rolled back. All SQL statements in a transaction are a whole, advancing and retreating together, and are inseparable. Either all are executed successfully or all fail. The primary key is the primary key, which is one or more fields in a table, and its value is used to uniquely identify a record in the table. In the relationship between two tables, the primary key is used to reference a specific record from another table in one table. The primary key is a unique keyword, part of the table definition. The primary key of a table can be composed of multiple keywords, and the column of the primary key cannot contain null values. The primary key is optional and can be defined in the CREATE TABLE or ALTER TABLE statement. Therefore, when a transaction occurs in the slave terminal 102 and the host server 104, it can be determined whether the transaction is a transaction without a primary key table or a transaction with a primary key table. Only when a transaction without a primary key table occurs, log aggregation processing is required to quickly apply logs.

[0096] Specifically, the log aggregation method of the present application is mainly used for processing transactions without primary key tables. When log processing is performed by the log aggregation method of the present application, the transaction type of the transaction can be judged first. Only when the transaction type of the transaction processing request is a transaction without a primary key table, the binary log corresponding to the transaction processing request can be obtained, thereby performing subsequent log aggregation processing. For transactions with primary key tables, the application process does not need to perform log aggregation processing. The method of the present application can remain open when the master and slave machines perform binary log communication. When there are large transactions with or without primary key tables (long running time, long-term uncommitted transactions), the logs of the transactions can be quickly aggregated to achieve the purpose of rapid application. In the present embodiment, the aggregation processing of binary logs is performed by identifying the transaction type of the transaction processing request, which can ensure the effectiveness of the log aggregation processing process, thereby ensuring the rapid progress of binary log communication between the master and slave machines.

[0097] In one embodiment, if Figure 5 As shown, step 207 includes:

[0098] Step 502: When the comparison result shows that the event meta information is inconsistent, the row format data of the event stored in the preset event buffer is obtained.

[0099] Step 504, concatenate the row format data of the stored events to obtain concatenated data.

[0100] Step 506, obtaining an aggregate event according to the spliced ​​data.

[0101] Step 508: Move the aggregated event out of the preset event buffer area, and clear the preset event buffer area.

[0102] Among them, row format data refers to the row-formatted log data generated by DML operations and stored in row format events. Aggregate events refer to an event obtained by aggregating multiple stored events in the preset event buffer. This event has the same event meta information as the stored events in the preset event buffer and contains the row format data of all stored events.

[0103] Specifically, when the comparison result is that the event meta-information is inconsistent, all the stored events in the preset event buffer area can be aggregated. When the stored events in the preset event buffer area are aggregated, the row format data of all stored events can be spliced ​​together to splice multiple row format events into one row format event, and ensure that the log data will not be lost. After obtaining the aggregated event, the aggregated event can be moved out of the preset event buffer area, thereby clearing the preset event buffer area. And after that, the aggregated event is executed using hash scanning. Thereby completing the complete process of row format event aggregation processing. In this embodiment, the aggregated event is obtained by splicing row format data, which can effectively ensure the completeness of the log data during the log aggregation processing and prevent the loss of log data.

[0104] In one embodiment, if Figure 6 As shown, step 506 specifically includes:

[0105] Step 601, obtaining the initial stored events in the preset event buffer area.

[0106] Step 603: Replace the row format data of the initial stored event with the spliced ​​data to obtain an aggregated event.

[0107] The initial stored event refers to the event that first enters the preset event buffer area.

[0108] Specifically, when obtaining aggregated events based on spliced ​​data, you can first identify the event that initially enters the preset event cache from multiple events in the preset event cache, and then replace the row format data of the initial stored event with the spliced ​​data, thereby replacing all the row format data in the initial stored event with the row format data of all events in the preset event cache. In this way, the initial stored event already contains the row format data of all stored events in the preset event cache, so it can be used to replace all events in the preset event cache, reducing the number of events in the event execution process, thereby improving the binary log communication efficiency between the master and slave machines. Figure 7 As shown, the preset event buffer contains event 1, event 2...event N, a total of N stored row format events. First, it is necessary to merge the row format data of all row format events in the preset event buffer, and then replace the initial stored event, that is, the row format data of event 1, with the merged row format data, thus completing the entire aggregation action. In the subsequent event execution process, hash scanning can be used to execute the aggregated events, so that only one full disk scan is required to complete the execution of N events, while before, this required N full disk scans. Aggregating row format logs through the log aggregation method of the present application can effectively speed up the log execution efficiency of the slave end, thereby improving the communication efficiency of binary log communication between the master and the slave, and reducing communication latency.

[0109] In one embodiment, after step 508, the method further includes: executing aggregate events and placing row format events into a preset event buffer.

[0110] Specifically, after obtaining the aggregated event, hash scanning can be used to execute the aggregated event. When executing the aggregated event, the newly extracted row format event can also be completely imported into the preset event buffer area whose content has been emptied, and used as a new initial stored event to continue the log aggregation processing of the subsequent process. When a new row format event is obtained, it is necessary to compare the obtained row format event with the new initial stored event to perform log aggregation processing. In this embodiment, by executing the aggregated event, the smooth progress of the binary log communication process is guaranteed, and by placing the row format event into the preset event buffer area, it is guaranteed that in the subsequent process, the aggregation process of the binary log can continue, and the efficiency of the binary log communication between the master and the slave is guaranteed.

[0111] The present application also provides an application scenario, which applies the above-mentioned log aggregation method. Specifically, the application of the log aggregation method in the application scenario is as follows:

[0112] The database system includes a master and a slave, the master is a node that generates binary logs, and the slave is a node that consumes binary logs. When a large transaction without a primary key table occurs between the master and the slave, binary log communication is required. In order to ensure the efficiency of applying binary logs on the slave side, the log aggregation method of the present application can be used to aggregate the binary logs sent by the master on the slave side, thereby ensuring the efficiency of log application. First, the host and the slave will obtain a transaction processing request; determine the transaction type of the transaction processing request; only when the transaction type of the transaction processing request is a transaction without a primary key table, the method of the present application will be used, and at this time it is necessary to obtain the binary log corresponding to the transaction processing request. The subsequent log aggregation processing process can refer to Figure 8As shown, first, it is necessary to read the binary log in units of events; and determine the event type of the read event; when the event type of the read event is a row format event type, extract the read event from the binary log and use the read event as a row format event. If it is not a row format event, execute it directly. Then determine whether the preset event buffer is empty. If it is empty (i.e., the initial stage of the solution), the row format event is directly placed in the preset event buffer. If it is not empty, it is necessary to compare the event meta information of the row format event with the stored event in the preset event buffer. When the comparison result shows that the event meta information is consistent, the row format event is directly placed in the preset event buffer. When the comparison result shows that the event meta-information is inconsistent, the stored events in the preset event buffer are aggregated. At this time, it is necessary to obtain the row format data of the stored events in the preset event buffer; splice the row format data of the stored events to obtain spliced ​​data; obtain the initial stored events in the preset event buffer; replace the row format data of the initial stored events with the spliced ​​data to obtain the aggregated event; move the aggregated event out of the preset event buffer, and clear the preset event buffer. Execute the aggregated event and place the row format event into the preset event buffer.

[0113] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] Based on the same inventive concept, the embodiment of the present application also provides a log aggregation device for implementing the log aggregation method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more log aggregation device embodiments provided below can refer to the limitations on the log aggregation method above, and will not be repeated here.

[0115] In one embodiment, Fig. 9 As shown, a log aggregation device is provided, including: a data acquisition module 902, an event extraction module 904, an event comparison module 906 and a log aggregation module 908, wherein:

[0116] The data acquisition module 902 is used to acquire binary logs.

[0117] The event extraction module 904 is used to extract row format events from the binary log.

[0118] The event comparison module 906 is used to compare the event metadata of the row format event with the events stored in the preset event buffer.

[0119] The log aggregation module 908 is used to place the row format event into a preset event buffer when the comparison result shows that the event meta information is consistent, and to aggregate the stored events in the preset event buffer when the comparison result shows that the event meta information is inconsistent.

[0120] In one embodiment, the event extraction module 904 is specifically used to: read events from the binary log in units of events; determine the event type of the read event; when the event type of the read event is a row format event type, extract the read event from the binary log and use the read event as a row format event.

[0121] In one embodiment, the data acquisition module 902 is specifically used to: acquire a transaction processing request; determine the transaction type of the transaction processing request; when the transaction type of the transaction processing request is a transaction without a primary key table, acquire a binary log corresponding to the transaction processing request.

[0122] In one embodiment, the log aggregation module 908 is specifically used to: when the comparison result is inconsistent event metadata, obtain the row format data of the events stored in the preset event cache; splice the row format data of the stored events to obtain spliced ​​data; obtain aggregated events based on the spliced ​​data; move the aggregated events out of the preset event cache, and clear the preset event cache.

[0123] In one embodiment, the log aggregation module 908 is specifically used to: obtain the initial stored event in the preset event buffer area; replace the row format data of the initial stored event with the spliced ​​data to obtain the aggregated event.

[0124] In one embodiment, the log aggregation module 908 is further used to: execute the aggregation event and place the row format event into the preset event buffer area

[0125] Each module in the above-mentioned log aggregation device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0126] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a log aggregation method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0127] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0129] Get the binary log.

[0130] Extract row-format events from the binary log.

[0131] Compare the event metadata of the row format event with the events stored in the preset event buffer.

[0132] When the comparison result shows that the event meta information is consistent, the row format event is placed in the preset event buffer area. When the comparison result shows that the event meta information is inconsistent, the stored events in the preset event buffer area are aggregated.

[0133] In one embodiment, when the processor executes the computer program, the following steps are further implemented: reading events from the binary log in units of events; determining the event type of the read event; when the event type of the read event is a row format event type, extracting the read event from the binary log and treating the read event as a row format event.

[0134] In one embodiment, when the processor executes the computer program, the following steps are further implemented: obtaining a transaction processing request; determining a transaction type of the transaction processing request; when the transaction type of the transaction processing request is a transaction without a primary key table, obtaining a binary log corresponding to the transaction processing request.

[0135] In one embodiment, when the processor executes the computer program, the following steps are also implemented: when the comparison result is that the event meta-information is inconsistent, the row format data of the events stored in the preset event cache are obtained; the row format data of the stored events are spliced ​​to obtain spliced ​​data; the aggregated events are obtained based on the spliced ​​data; the aggregated events are moved out of the preset event cache, and the preset event cache is cleared.

[0136] In one embodiment, when the processor executes the computer program, the following steps are further implemented: obtaining the initial stored event in the preset event buffer; replacing the row format data of the initial stored event with the spliced ​​data to obtain the aggregated event.

[0137] In one embodiment, when the processor executes the computer program, the following steps are further implemented: executing the aggregation event, and placing the row format event into a preset event buffer.

[0138] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0139] Get binary log. Extract row-format events from binary log.

[0140] Compare the event metadata of the row format event with the events stored in the preset event buffer.

[0141] When the comparison result shows that the event meta information is consistent, the row format event is placed in the preset event buffer area. When the comparison result shows that the event meta information is inconsistent, the stored events in the preset event buffer area are aggregated.

[0142] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: reading events from the binary log in units of events; determining the event type of the read event; when the event type of the read event is a row format event type, extracting the read event from the binary log and treating the read event as a row format event.

[0143] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining a transaction processing request; determining the transaction type of the transaction processing request; when the transaction type of the transaction processing request is a transaction without a primary key table, obtaining a binary log corresponding to the transaction processing request.

[0144] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the comparison result is that the event meta-information is inconsistent, the row format data of the events stored in the preset event cache are obtained; the row format data of the stored events are spliced ​​to obtain spliced ​​data; the aggregated events are obtained based on the spliced ​​data; the aggregated events are moved out of the preset event cache, and the preset event cache is cleared.

[0145] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining the initial stored event in the preset event buffer; replacing the row format data of the initial stored event with the spliced ​​data to obtain the aggregated event.

[0146] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: executing the aggregation event, and placing the row format event into a preset event buffer.

[0147] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0148] Get the binary log.

[0149] Extract row-format events from the binary log.

[0150] Compare the event metadata of the row format event with the events stored in the preset event buffer.

[0151] When the comparison result shows that the event meta information is consistent, the row format event is placed in the preset event buffer area. When the comparison result shows that the event meta information is inconsistent, the stored events in the preset event buffer area are aggregated.

[0152] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: reading events from the binary log in units of events; determining the event type of the read event; when the event type of the read event is a row format event type, extracting the read event from the binary log and treating the read event as a row format event.

[0153] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining a transaction processing request; determining the transaction type of the transaction processing request; when the transaction type of the transaction processing request is a transaction without a primary key table, obtaining a binary log corresponding to the transaction processing request.

[0154] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the comparison result is that the event meta-information is inconsistent, the row format data of the events stored in the preset event cache are obtained; the row format data of the stored events are spliced ​​to obtain spliced ​​data; the aggregated events are obtained based on the spliced ​​data; the aggregated events are moved out of the preset event cache, and the preset event cache is cleared.

[0155] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining the initial stored event in the preset event buffer; replacing the row format data of the initial stored event with the spliced ​​data to obtain the aggregated event.

[0156] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: executing the aggregation event, and placing the row format event into a preset event buffer.

[0157] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A log aggregation method, characterized in that: The method comprises: Get binary logs; Extracting row format events in the binary log; Comparing the row format event with the events stored in the preset event buffer for event meta information; When the comparison result shows that the event meta information is consistent, the row format event is placed in the preset event cache area; when the comparison result shows that the event meta information is inconsistent, the stored events in the preset event cache area are aggregated; the event meta information includes the type of event, the table operated by the event, and the unique code of the server where the event is generated.

2. The method according to claim 1, characterized in that The extracting the row format events in the log comprises: Reading events from the binary log in units of events; Determine the event type of the event being read; When the event type of the read event is a row format event type, the read event is extracted from the binary log, and the read event is used as a row format event.

3. The method according to claim 1, characterized in that Obtaining the binary log includes: Get transaction request; Determining a transaction type of the transaction processing request; When the transaction type of the transaction processing request is a transaction without a primary key table, a binary log corresponding to the transaction processing request is obtained.

4. The method according to claim 1, characterized in that: When the comparison result shows that the event meta information is inconsistent, aggregating the stored events in the preset event buffer area includes: When the comparison result shows that the event meta information is inconsistent, the row format data of the event stored in the preset event buffer is obtained; splicing the row format data of the stored events to obtain spliced ​​data; Acquire an aggregation event according to the spliced ​​data; The aggregated event is moved out of the preset event buffer area, and the preset event buffer area is cleared.

5. The method according to claim 4, characterized in that The obtaining of an aggregate event according to the spliced ​​data comprises: Obtaining the initial stored events in the preset event buffer; The row format data of the initial stored event is replaced with the spliced ​​data to obtain an aggregated event.

6. The method according to claim 4, characterized in that After the aggregate event is moved out of the preset event buffer area and the preset event buffer area is cleared, the method further includes: The aggregate event is executed, and the row format event is placed into the preset event buffer.

7. A log aggregation device, characterized in that: The device comprises: Data acquisition module, used to obtain binary logs; An event extraction module, used to extract row format events from the binary log; An event comparison module, used for comparing event meta information of the row format event with events stored in a preset event buffer; The log aggregation module is used to place the row format event into the preset event buffer when the comparison result shows that the event meta information is consistent, and to aggregate the stored events in the preset event buffer when the comparison result shows that the event meta information is inconsistent; the event meta information includes the type of event, the table operated by the event, and the unique code of the server where the event is generated.

8. The device according to claim 7, characterized in that The event extraction module is specifically used to: read events from the binary log in units of events; determine the event type of the read event; when the event type of the read event is a row format event type, extract the read event from the binary log and use the read event as a row format event.

9. The device according to claim 8, characterized in that The data acquisition module is specifically used to: acquire a transaction processing request; determine the transaction type of the transaction processing request; when the transaction type of the transaction processing request is a transaction without a primary key table, acquire a binary log corresponding to the transaction processing request.

10. The device according to claim 8, characterized in that The log aggregation module is specifically used to: when the comparison result shows that the event meta information is inconsistent, obtain the row format data of the event stored in the preset event buffer area; splice the row format data of the stored event to obtain spliced ​​data; Acquire an aggregation event according to the spliced ​​data; The aggregated event is moved out of the preset event buffer area, and the preset event buffer area is cleared.

11. The device according to claim 10, characterized in that The log aggregation module is also used to: obtain the initial stored event in the preset event buffer area; replace the row format data of the initial stored event with the spliced ​​data to obtain the aggregated event.

12. The device according to claim 10, characterized in that The log aggregation module is further used to: execute the aggregation event and place the row format event into the preset event buffer area.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images