Log processing method, system, device, client and storage medium

Abstract

Embodiments of the present application disclose a log processing method, system, device, client and storage medium. The method is applied to a log processing client, and the method comprises: obtaining a change operation log based on middleware, wherein the change operation log is a corresponding log sent by a log obtaining server when the log obtaining server listens to a pre-set binary log of a relational database based on the middleware to make a log change; filtering target message bodies belonging to the same transaction from the change operation log according to the types of each message body in the change operation log, processing the target message bodies, and storing processing results in a log database, wherein the processing results comprise data before the change operation and data after the change operation. The technical scheme of the embodiments of the present application achieves the effect of simultaneously obtaining the data before the change operation and the data after the change operation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer application technology, and in particular to a log processing method, system, device, client, and storage medium. Background Art

[0002] Currently, most major systems use MySQL to manage data. For example, the MySQL database corresponding to the product system stores hundreds of millions of data points, including product information, category information, and more. Faced with such a large amount of data, the product log processing system came into being to clearly record the user's operation process on this data.

[0003] The architecture of existing commodity log systems is the ELK (Elasticsearch + Logstash + Kibana) architecture. Logstash acts as a pipeline to read log data from MySQL and stores the results in Elasticsearch. Kibana is a web-based graphical interface that can visualize the log data stored in Elasticsearch.

[0004] In the process of implementing the present invention, the inventors discovered that the following technical problems exist in the prior art: the existing commodity log processing system cannot obtain data before the change operation, and the user experience is poor. Summary of the Invention

[0005] The embodiments of the present invention provide a log processing method, system, device, client and storage medium, which solve the problem that the log processing system cannot obtain data before the change operation.

[0006] In a first aspect, an embodiment of the present invention provides a log processing method, which is applied to a log processing client. The method may include:

[0007] Obtaining change operation logs based on the middleware, wherein the change operation logs are corresponding logs sent by the log acquisition server when the middleware monitors the binary logs of the preset relational database for log changes;

[0008] According to the type of each message body in the change operation log, the target message body belonging to the same transaction is filtered out from the change operation log, the target message body is processed, and the processing result is stored in the log database. The processing result includes the data before the change operation and the data after the change operation.

[0009] In a second aspect, an embodiment of the present invention further provides a log processing system, which may include: a log database, and a log processing client and a log acquisition server configured with middleware, wherein the log processing client is used to execute the log processing method provided by any embodiment of the present invention.

[0010] In a third aspect, an embodiment of the present invention further provides a log processing device, which is configured on a log processing client. The device may include:

[0011] A log acquisition module is used to acquire change operation logs based on the middleware, wherein the change operation logs are corresponding logs sent by the log acquisition server when the middleware monitors the binary logs of the preset relational database for log changes;

[0012] The log processing module is used to filter out the target message bodies belonging to the same transaction from the change operation log according to the type of each message body in the change operation log, process the target message body, and store the processing results in the log database. The processing results include the data before the change operation and the data after the change operation.

[0013] In a fourth aspect, an embodiment of the present invention further provides a log processing client, which may include:

[0014] one or more processors;

[0015] a memory for storing one or more programs;

[0016] When one or more programs are executed by one or more processors, the one or more processors implement the log processing method provided by any embodiment of the present invention.

[0017] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the log processing method provided by any embodiment of the present invention.

[0018] The technical solution of the embodiment of the present invention is that after obtaining the change operation log through the middleware, the target message body belonging to the same transaction can be filtered out from the change operation log according to the type of each message body in the change operation log, and the target message body can be processed, thereby respectively obtaining the data before the change operation and the data after the change operation, wherein the above-mentioned change operation log can be the corresponding log sent after the log acquisition server obtains the log change based on the middleware monitoring the binary log of the preset relational database. The above-mentioned technical solution, based on the middleware obtaining the change operation log in which the log change occurs from the binary log, and processing the target message body belonging to the same transaction filtered out from the change operation log, achieves the effect of simultaneously obtaining the data before and after the change operation, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of a log processing method in Embodiment 1 of the present invention;

[0020] Figure 2 This is a structural diagram of a log processing system in Embodiment 2 of the present invention;

[0021] Figure 3 This is a structural diagram of a log processing system in Embodiment 3 of the present invention;

[0022] Figure 4 This is a schematic diagram of an application of a log processing system in Embodiment 3 of the present invention;

[0023] Figure 5 This is a structural block diagram of a log processing device in a fourth embodiment of the present invention;

[0024] Figure 6 This is a structural diagram of a log processing client in Example 5 of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0026] Example 1

[0027] Figure 1 This is a flow chart of a log processing method provided in the first embodiment of the present invention, which can be applied to a log processing client. This embodiment is applicable to processing acquired change operation logs to obtain data before and after the change operation. This method can be performed by the log processing device provided in this embodiment of the present invention, which can be implemented in software and / or hardware.

[0028] See also Figure 1 The method of the embodiment of the present invention specifically includes the following steps:

[0029] S110 . Obtain a change operation log based on the middleware, wherein the change operation log is a corresponding log sent by the log acquisition server when the middleware monitors a preset binary log of a relational database for log changes.

[0030] A relational database is a database built on a relational model, such as MySQL, SQL Server, Oracle, Access, InterBase, and others. The relevant information involved in performing changes to data stored in a relational database can be recorded in a binary log (binlog), a binary log file. This information can include data before and after the change, the name of the operation database, the name of the operation table, the table type, the table primary key, the operation type, the operation time, the operation source, the operator, and the operation IP address. The operation database name is the name of the database corresponding to the change operation, the operation table name is the name of the table corresponding to the change operation, the table type is the table type (e.g., product table, category (SKU) table), the table primary key is the business primary key (ID), and the operation type can be update, delete, insert, and so on. Optionally, the operator and the operation IP address can be recorded in the binlog or op_log table. If their recording paths are different, the log processing system can read the operator and operation IP address from the corresponding recording path.

[0031] The log acquisition server can be configured with the address of the database and its corresponding table that needs to record the change operation log, so that the corresponding binary log can be monitored according to the address. Therefore, when the log acquisition server monitors the binary log of the pre-set relational database based on the middleware to make a log change, it can obtain the log change operation log from the binary log and send the change operation log to the log processing client that is also configured with the middleware. Optionally, when the database and / or table that the log acquisition server needs to record the change operation log is added and / or deleted, the log acquisition server can be configured accordingly so that the log acquisition server monitors the binlog corresponding to the newly added database and / or table, and / or no longer monitors the binlog corresponding to the deleted database and / or table.

[0032] For example, the log acquisition server's reading process is described using MySQL as a relational database and Canal as the middleware. MySQL operates in a master-slave relationship: the master records change operation logs in the binlog. These records are called binary log events and can be viewed using the command "show binlog events." The slave copies the master's binary log events to its own relay log. The slave redoes the relay log events to reflect its own data in the change operation log. Thus, Canal can simulate the interaction protocol of MySQL's slaves, disguising itself as a slave to send a dump request to the MySQL master. After receiving the dump request, the master pushes the change operation log in the binlog to Canal, allowing the log acquisition server configured with Canal to obtain the change operation log.

[0033] S120. Filter target message bodies belonging to the same transaction from the change operation log according to the type of each message body in the change operation log, process the target message body, and store the processing result in the log database, wherein the processing result includes the data before the change operation and the data after the change operation.

[0034] Among them, since the change operation log may involve multiple transactions, in order to ensure that the subsequent log processing process is processed for a complete transaction, the log processing client can filter out the target message bodies belonging to the same transaction from the change operation log according to the type of each message body in the change operation log after receiving the change operation log sent by the log acquisition server. For example, the type can be transaction start, data before and after the change operation, and transaction end, that is, the message body of transaction start, the message body of data before and after the change operation, and the message body of transaction end can constitute a complete transaction, and these message bodies are target message bodies belonging to the same transaction.

[0035] Furthermore, the target message body is processed, that is, each message body belonging to the same transaction is processed together, thereby avoiding the situation where half of the change operation is recorded. Among them, the above processing process can be selected according to the actual business scenario. For example, if the change operation log includes multiple fields, the fields in the multiple fields that are not related to the log processing system can be filtered. For example, the fields that are not related to the log processing system can be those data that have not undergone change operations, such as multiple fields are keyVlue^keyVlue^keyVlue^…, each keyVlue includes data before the change operation and data after the change operation, thereby analyzing the inside of each keyVlue to filter out the fields where the data where the change operation has not occurred is located, that is, only retaining the fields where the data where the change operation has occurred is located; for example, special fields in the multiple fields can be specially processed and general fields can be generally processed. The special processing can be a processing logic customized according to the specific business scenario; and so on. On this basis, the target message body (i.e., the processing result) after the above processing process can include the data before the change operation, the data after the change operation, the operation library name, the operation table name, the table type, the table primary key, the operation type, the operation time, the operation source, the operator, the operation IP and other information.

[0036] Furthermore, the processing results can be sent to a log database to be stored in the log database. The log database can be Elasticsearch, Hbase, etc., wherein Hbase is a database that supports massive data storage. Taking Hbase as an example, the above storage process can be to encapsulate the processing results into an object containing information such as data before the change operation, data after the change operation, operation library name, operation table name, table type, table primary key, operation type, operation time, operation source, operator, operation IP, etc., and store the object in Hbase. Optionally, assuming that the number of fields where the data of the change operation appears in a certain target message body is at least two, these fields can be encapsulated into different objects to be stored separately in Hbase, thereby improving the clarity of data storage.

[0037] The technical solution of the embodiment of the present invention is that after obtaining the change operation log through the middleware, the target message body belonging to the same transaction can be filtered out from the change operation log according to the type of each message body in the change operation log, and the target message body can be processed, thereby respectively obtaining the data before the change operation and the data after the change operation, wherein the above-mentioned change operation log can be the corresponding log sent after the log acquisition server obtains the log change based on the middleware monitoring the binary log of the preset relational database. The above-mentioned technical solution, based on the middleware obtaining the change operation log in which the log change occurs from the binary log, and processing the target message body belonging to the same transaction filtered out from the change operation log, achieves the effect of simultaneously obtaining the data before and after the change operation, thereby improving the user experience.

[0038] An optional technical solution, in which the table information of each table in the relational database is encapsulated in a common package and the common package is configured in the log processing client, the table information includes the table name, and the target message body is processed, which can specifically include: filtering out the target table name corresponding to the target message body from each table name, and processing the target message body based on the processing strategy corresponding to the target table name recorded in the common package.

[0039] Among them, in order to make the relational database and the tables therein flexible and scalable, the relevant information of the table can be encapsulated in an independent common package, such as encapsulating the relevant information of various relational databases and various tables in each relational database in a common package, and directly introducing the common package on the log processing client, that is, configuring the common package on the log processing client, at this time, there is no need to redefine the relevant information of the table on the log processing client, and the relevant information of the table can be processed through the common, that is, when it is necessary to add and / or delete tables, or add, change and / or delete the relevant information of the table, the corresponding changes can be made directly in the conmon package, and the common package can be reintroduced on the log processing client without the need to modify the code on the log processing client. The above-mentioned common package is equivalent to a toolkit, which can include information such as table name (i.e., table type), table number, table primary key, table field and description.

[0040] On this basis, when processing the target message body, you can first filter out the target table name corresponding to the target message body from the table names in the common package. For example, the target message body may contain its corresponding table name, but it is impossible to determine what kind of table the corresponding table is based on the table name of the target message body. At this time, you can determine what kind of table it is based on the table names in the common package that have been introduced, that is, determine what kind of table it is based on the target table name in the common package corresponding to the table name of the target message body; thus, you can process the target message body according to the processing strategy corresponding to such a table, so that some tables that require special processing can be specially processed according to custom logic.

[0041] An optional technical solution, the log database includes Hbase; storing the processing results in the log database, specifically may include: determining the row key calculation factor based on the processing result, and calculating the row key based on the row key calculation factor, and storing the processing result in the storage interval corresponding to the row key in Hbase. Among them, Hbase is a nosql (not only sql) database. In the process of adding, deleting, modifying and checking, the row key (rowkey) plays the role of the primary key, which can uniquely identify a row of data. The row key calculation factor is a factor used to calculate the rowkey. Its specific selection and the specific process of calculating the rowkey based on the row key calculation factor can be determined according to the actual business scenario. Exemplarily, the row key calculation factor may include at least one of the table type, table primary key, operation timestamp and operation version number of the target message body. According to the row key calculation factor, a rowkey rule can be pre-set to calculate the rowkey. Thus, the processing result can be stored in the storage interval corresponding to the rowkey in Hbase. For example, the row key calculation factors include table type, inverted table primary key, 64-bit signed maximum value, operation timestamp and operation version number. The 64-bit signed maximum value is a preset value, the operation timestamp is the timestamp corresponding to the operation time of updating the operation log, and the operation version number is the version number corresponding to the object when the processing result is encapsulated as an object. The rowkey generated based on the pre-set rowkey rule is composed of a binary array according to the table type + inverted business primary key + (64-bit signed maximum value - operation timestamp - operation version number). On this basis, optionally, in order to disperse different processing results to different storage intervals of Hbase, after calculating the rowkey, its hash value can be recalculated, and the processing result can be stored in the storage interval corresponding to the hash value in Hbase, thereby avoiding the emergence of hot data.

[0042] An optional technical solution, the above-mentioned log processing method may also include: reading the operation time from the distributed service, comparing the operation time with the current time, and feeding back the comparison result, wherein the operation time is the time when the log acquisition server connected to the distributed service obtains the change operation log and then writes it to the distributed service. The distributed service may be a distributed, open source distributed application coordination service, which can encapsulate complex and error-prone key services and provide a simple and easy-to-use interface to user applications; the operation time is the time when the change operation log appears, and the current time is the time when the log processing client executes the above steps. Optionally, the above-mentioned distributed service may be Zookeeper.

[0043] Considering the possible application scenarios involved in the embodiments of the present invention, the distributed service can be used to record the operation location of the relational database, which is the operation time of the change operation log. Therefore, when the log acquisition server and the log processing client are both configured with the address of the distributed service to access the registration center of the distributed service, the log acquisition server can obtain the operation time from the change operation log and write the operation time to the distributed service; further, the log processing client can obtain the operation time from the distributed service and perform other operations based on the comparison result of the operation time and the current time. For example, based on the comparison result, it can determine whether certain technologies need to be optimized, monitor whether there are any abnormalities in the current working status of the log processing system, etc.

[0044] For example, when a log processing client processes a change operation log, it can compare the operation time with the current time. If the difference between the two is large, it indicates that the change operation log was not processed in a timely manner. This may be due to the log acquisition server not obtaining the change operation log in a timely manner, the log processing client having too many change operation logs to process, and the change operation log currently being processed not being processed in a timely manner. For another example, when the log processing client stores the processing result in Hbase, it can compare the operation time with the current time. If the difference between the two is large, it indicates that the change operation log was not stored in a timely manner. This may be due to the log acquisition server not obtaining the change operation log in a timely manner, the log processing client having too many change operation logs to process, and the change operation log currently being processed not being processed in a timely manner. In other words, when a delay is determined based on the comparison results, the cause can be investigated to optimize the technology and determine whether there is an abnormality in the current working state of the log processing system.

[0045] Example 2

[0046] Before introducing Example 2, we first analyze the ELK architecture in the prior art in detail again: First, Logstash consumes a lot of resources during operation due to its high CPU and memory usage; in addition, it reads log data from the MySQL table. This reading process will put a lot of pressure on MySQL, and the read result is only the data after the operation update (i.e., incremental update), and it cannot directly operate on the read result. Secondly, Elasticsearch can provide the three major functions of collecting, analyzing and storing log data. It is more suitable for application scenarios with more read operations, while log processing systems mostly involve write operations, and read operations mostly appear in a small number of scenarios such as troubleshooting. Thirdly, Kibana's visualization function is even more insignificant in log processing systems that mainly record log data.

[0047] Accordingly, Figure 2 This is a structural diagram of a log processing system provided in the second embodiment of the present invention. This embodiment is applicable to recording change operations in a relational database. Figure 2 The log processing system of the embodiment of the present invention may include:

[0048] A log acquisition server 20 configured with middleware, a log processing server 21 configured with middleware, and a log database 22, wherein the log acquisition server 20 is used to obtain the corresponding change operation log when the binary log of a preset relational database is monitored by the middleware, and send the change operation log to the log processing server 21; the log processing server 21 is used to filter out the target message body belonging to the same transaction from the change operation log according to the type of each message body in the change operation log, and process the target message body, and store the processing result in the log database 22, wherein the processing result includes the data before the change operation and the data after the change operation.

[0049] It should be noted that the log processing system described in this embodiment of the present invention uses middleware to read change operation logs from binlog. This allows for near-real-time synchronization of change operation logs, and even in the face of massive amounts of data, it does not put pressure on the relational database and impact online business. The read results can also be directly manipulated to obtain the data before and after the change operation, which is essential and critical data in the log processing system. As a result, the log processing system can accurately and in real time record change operations related to these massive amounts of data, improving the performance and stability of the log processing system while reducing resource consumption costs.

[0050] The technical solution of the embodiment of the present invention is to obtain the corresponding change operation log when the log change is made to the binary log of a pre-set relational database monitored by the log acquisition server based on the middleware, and send the change operation log to the log processing client configured with the middleware; the log processing client can filter out the target message body belonging to the same transaction from the type of each message body in the change operation log, and process the target message body, and store the processing result in the log database. The processing result may include the data before the change operation and the data after the change operation. The above technical solution, based on the middleware, obtains the change operation log where the log change occurs from the binary log, and processes the target message body belonging to the same transaction filtered out from the change operation log, thereby achieving the effect of simultaneously obtaining the data before and after the change operation, thereby improving the user experience.

[0051] Example 3

[0052] Figure 3 This is a schematic diagram of the structure of a log processing system provided in Example 3 of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, the log processing system may optionally further include a log search server for searching the log database based on search criteria received from the user and returning target data that matches the search criteria. The explanations of terms that are identical or corresponding to those in the above-mentioned embodiments are not repeated here.

[0053] See also Figure 3 The log processing system of this embodiment may include: a log acquisition server 30 configured with middleware, a log processing client 31 configured with middleware, a log database 32, and a log search server 33, wherein the log acquisition server 30 is used to obtain the corresponding change operation log when the binary log of the preset relational database is monitored by the middleware, and send the change operation log to the log processing client 31; the log processing client 31 is used to filter out the target message body belonging to the same transaction from the change operation log according to the type of each message body in the change operation log, and process the target message body, and store the processing result in the log database 32, wherein the processing result includes the data before the change operation and the data after the change operation; the log search server 33 is used to search in the log database 32 according to the search conditions input by the user, and return the target data that matches the search conditions.

[0054] Among them, the log search server 33 can provide a front-end page so that the user can enter the search conditions in the front-end page. Therefore, after receiving the search conditions entered by the user, it can search in the log database 32 according to the search conditions to search for target data that matches the search conditions, and return the target data to the front-end page to display to the user. Exemplarily, considering the application scenarios that may be involved in the embodiments of the present invention, the front-end page (i.e., web page) can be implemented based on vm and / or js, which can realize the selection function and / or input function of the table type, table primary key, operation time period, keyword and other contents. Therefore, the log search server 33 can form a search condition based on the user's selection content and / or input content, and search in the log database 32 according to the search condition to return the target data that matches the search condition.

[0055] It should be noted that the search results generated by the search function provided by Elasticsearch are strings containing various types of data. Such search results are difficult for users to read, and it is difficult for users to find correlations among the various data types stacked together. Accordingly, the target data searched by the log search server described in the embodiment of the present invention can present the various types of data separately, such as placing the data before the change operation on the left side of the page and the data after the change operation on the right side of the page, thereby improving the readability of the search results.

[0056] The technical solution of the embodiment of the present invention receives the search conditions input by the user through the log search server, searches for matching target data in the log database based on the search conditions and returns them, thereby achieving the effect of fast and accurate search of the data stored in the log database.

[0057] On this basis, optionally, the log database includes Hbase and a log search server, which is specifically used to calculate the row key based on the search conditions received from the user input, and return the target data in the storage interval corresponding to the row key in Hbase. Among them, as mentioned above, when the log database is Hbase, the specific storage interval it stores is related to the calculated rowkey. Therefore, when searching in Hbase according to the search conditions, the rowkey can also be calculated according to the search conditions first, and then the data stored in the storage interval corresponding to the rowkey in Hbase is used as the target data. It should be noted that if the operation time in the search conditions is a time period, a rowkey can be calculated according to the start time and end time of the time period, and the data in the storage interval where the two rowkeys are located and the storage interval between the two storage intervals are used as the target data.

[0058] Furthermore, optionally, the search conditions include keywords, the table information of each table in the relational database is encapsulated in a common package and the common package is configured in the log search server, and the table information includes the mapping relationship between each attribute information of the table; the log search server is also used to filter the target data according to the keywords and the mapping relationship, and update the target data according to the filtering results. Among them, the data in the storage interval where the rowkey is located is not necessarily the target data required by the user. In this case, it can be filtered again by the keyword entered by the user. It should be noted that each data can be composed of multiple fields, and the field where the keyword is located may be only one of these multiple fields. Therefore, in order to determine the remaining corresponding fields based on the keyword field, the mapping relationship between the attribute information of each table can be encapsulated in a common package, and the common package can be configured in the log search server, so that the log search server can determine the remaining attribute information that matches the keyword based on the mapping relationship in the common package. Each attribute information can occupy at least one field. For example, the table type and table number can be mapped into a HashMap, the table type and table primary key can be mapped into a HashMap, the table number and table primary key can be mapped into a HashMap, and the table type and table description information can be mapped into a HashMap. For example, if the keyword is the table number, the remaining attribute information corresponding to the table number can be determined based on the above HashMap, and the search results can be further filtered based on this attribute information.

[0059] In order to better understand the specific working process of the above-mentioned log processing system, the log processing system of this embodiment is exemplarily described below with reference to specific examples. Figure 4 As shown, the log processing system may include canalserver (i.e., a log acquisition server configured with the middleware canal), canalclient (i.e., a log processing client configured with the middleware canal), Log Search Web (i.e., a log search server providing a front-end page), and Hbase, wherein the canal server crawls the change operation log from MySQL based on canal and sends the crawling result to the canalclient; further, the canal client performs a series of processing such as analysis and filtering on the received change operation log, and stores the processing result in Hbase; thus, the subsequent Log Search Web can find matching target data in Hbase according to the search conditions entered by the user, thereby achieving the effect of real-time acquisition and accurate search of the change operation log.

[0060] Example 4

[0061] Figure 5 This is a structural block diagram of the log processing device provided in the fourth embodiment of the present invention. The device can be configured on the log processing client and is used to execute the log processing method provided in the first embodiment above. The device and the log processing method in the first embodiment above belong to the same inventive concept. For details not fully described in the embodiment of the log processing device, please refer to the embodiment of the log processing method above. Figure 5 , the device may specifically include: a log acquisition module 410 and a log processing module 420.

[0062] The log acquisition module 410 is used to acquire the change operation log based on the middleware, wherein the change operation log is the corresponding log sent by the log acquisition server when the middleware monitors the binary log of the preset relational database to perform log changes;

[0063] The log processing module 420 is used to filter out the target message bodies belonging to the same transaction from the change operation log according to the type of each message body in the change operation log, process the target message body, and store the processing results in the log database, where the processing results include the data before the change operation and the data after the change operation.

[0064] Optionally, table information of each table in the relational database is encapsulated in a common package and the common package is configured in the log processing client. The table information includes the table name.

[0065] Accordingly, the log processing module 420 may specifically include:

[0066] The log processing unit is used to filter out the target table name corresponding to the target message body from the various table names, and process the target message body based on the processing strategy corresponding to the target table name recorded in the common package.

[0067] Optionally, the log database includes Hbase, and the log processing module 420 may specifically include:

[0068] The log storage unit is used to determine the row key calculation factor based on the processing result, calculate the row key based on the row key calculation factor, and store the processing result in the storage interval corresponding to the row key in Hbase;

[0069] The row key calculation factors include at least one of the table type, table primary key, operation timestamp, and operation version number of the target message body.

[0070] Optionally, the log storage unit can also be used to recalculate the hash value of the row key and update the row key based on the calculation result.

[0071] Optionally, based on the above device, the device may further include:

[0072] The time comparison module is used to read the operation time from the distributed service, compare the operation time with the current time, and feedback the comparison result. The operation time is the time when the log acquisition server connected to the distributed service obtains the change operation log and then writes it to the distributed service.

[0073] The log processing device provided in the fourth embodiment of the present invention has a log acquisition module and a log processing module that cooperate with each other. After obtaining the change operation log through the middleware, the target message body belonging to the same transaction can be filtered out from the change operation log according to the type of each message body in the change operation log, and the target message body is processed, thereby respectively obtaining the data before the change operation and the data after the change operation. The above-mentioned change operation log can be the corresponding log sent after the log acquisition server obtains and sends it when the log change is performed based on the middleware monitoring the binary log of a preset relational database. The above-mentioned device obtains the change operation log of the log change from the binary log based on the middleware, and processes the target message body belonging to the same transaction filtered out from the change operation log, thereby achieving the effect of simultaneously obtaining the data before and after the change operation, thereby improving the user experience.

[0074] The log processing device provided by the embodiment of the present invention can execute the log processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0075] It is worth noting that in the embodiment of the above-mentioned log processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0076] Example 5

[0077] Figure 6 A schematic diagram of the structure of a log processing client provided in the fifth embodiment of the present invention is shown as follows: Figure 6 As shown, the log processing client includes a memory 510, a processor 520, an input device 530 and an output device 540. The number of processors 520 in the log processing client can be one or more. Figure 6 In the example, a processor 520 is used; the memory 510, the processor 520, the input device 530 and the output device 540 in the log processing client can be connected via a bus or other means. Figure 6 The connection via bus 550 is taken as an example.

[0078] The memory 510, as a computer-readable storage medium, can be used to store software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the log processing method in the embodiments of the present invention (for example, the log acquisition module 410 and the log processing module 420 in the log processing device). The processor 520 executes the various functional applications and data processing of the log processing client by running the software programs, instructions, and modules stored in the memory 510, thereby implementing the above-mentioned log processing method.

[0079] The memory 510 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the log processing client, etc. In addition, the memory 510 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 510 may further include a memory remotely located relative to the processor 520, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0080] The input device 530 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 540 may include a display device such as a display screen.

[0081] Example 6

[0082] A sixth embodiment of the present invention provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform a log processing method. The method is applied to a log processing client. The method may include:

[0083] Obtaining change operation logs based on the middleware, wherein the change operation logs are corresponding logs sent by the log acquisition server when the middleware monitors the binary logs of the preset relational database for log changes;

[0084] According to the type of each message body in the change operation log, the target message body belonging to the same transaction is filtered out from the change operation log, the target message body is processed, and the processing result is stored in the log database, where the processing result includes the data before the change operation and the data after the change operation.

[0085] Of course, the computer executable instructions of the storage medium including computer executable instructions provided by the embodiment of the present invention are not limited to the operations of the method described above, and can also execute related operations in the log processing method provided by any embodiment of the present invention.

[0086] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a computer's floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0087] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A log processing method, characterized in that: Applied to a log processing client, the log processing client is configured with middleware and connected to a distributed service, the method includes: Obtaining a change operation log based on the middleware, wherein the change operation log is a corresponding log sent by a log acquisition server when a log change is performed based on a binary log of a preset relational database monitored by the middleware, and the log acquisition server is configured with the middleware; Filtering target message bodies belonging to the same transaction from the change operation log according to the type of each message body in the change operation log, processing the target message bodies, and storing the processing results in a log database, the processing results including data before the change operation and data after the change operation, wherein each message body represented by the target message body constitutes a complete transaction; The operation time is read from the distributed service, the operation time is compared with the current time, and the comparison result is fed back. If a delay is determined according to the comparison result, the cause is investigated to perform an operation corresponding to the investigated cause, wherein the operation time is the time that the log acquisition server connected to the distributed service obtains the change operation log and then writes it into the distributed service.

2. The method according to claim 1, characterized in that Table information of each table in the relational database is encapsulated in a common package, and the common package is configured in the log processing client. The table information includes a table name. The processing of the target message body includes: A target table name corresponding to the target message body is screened out from the table names, and the target message body is processed based on a processing strategy corresponding to the target table name recorded in the common package.

3. The method according to claim 1, characterized in that The log database includes Hbase, and storing the processing results in the log database includes: Determine a row key calculation factor according to the processing result, calculate a row key according to the row key calculation factor, and store the processing result in a storage interval corresponding to the row key in Hbase; The row key calculation factor includes at least one of the table type, table primary key, operation timestamp, and operation version number of the target message body.

4. The method according to claim 3, characterized in that After calculating the row key according to the row key calculation factor, the method further includes: Recalculate the hash value of the row key and update the row key according to the calculation result.

5. A log processing system, characterized in that: include: A log database, a log processing client and a log acquisition server configured with middleware, wherein the log processing client is used to execute the log processing method according to any one of claims 1 to 4.

6. The system according to claim 5, characterized in that Also includes: The log search server is used to search the log database according to the search conditions input by the received user, and return the target data that matches the search conditions.

7. The system according to claim 6, characterized in that The log database includes Hbase; The log search server is specifically configured to calculate a row key according to a search condition received from a user, and return target data in a storage interval corresponding to the row key in Hbase.

8. The system according to claim 7, characterized in that The search condition includes a keyword, table information of each table in the relational database is encapsulated in a common package and the common package is configured in the log search server, and the table information includes a mapping relationship between each attribute information of the table; The log search server is further configured to filter the target data according to the keyword and the mapping relationship, and update the target data according to the filtering result.

9. A log processing device, characterized in that: Configured on a log processing client, the log processing client is configured with middleware and connected to a distributed service, the device includes: a log acquisition module, configured to acquire a change operation log based on the middleware, wherein the change operation log is a corresponding log sent by a log acquisition server when a log change is performed based on a binary log of a preset relational database monitored by the middleware, and the log acquisition server is configured with the middleware; a log processing module, configured to filter target message bodies belonging to the same transaction from the change operation log according to the type of each message body in the change operation log, process the target message bodies, and store the processing results in a log database, wherein the processing results include data before and after the change operation, and each message body represented by the target message body constitutes a complete transaction; A time comparison module is used to read the operation time from the distributed service, compare the operation time with the current time, and feed back the comparison result, so as to conduct a cause investigation when it is determined that there is a delay phenomenon according to the comparison result, and to perform an operation corresponding to the found cause, wherein the operation time is the time that the log acquisition server connected to the distributed service obtains from the change operation log and then writes it to the distributed service.

10. A log processing client, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log processing method according to any one of claims 1 to 4.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the log processing method according to any one of claims 1 to 4 is implemented.

Images