Unlock instant, AI-driven research and patent intelligence for your innovation.

Log processing method, system and device, client and storage medium

A processing method and client-side technology, applied in the file system, file system function, file system type, etc., can solve the problem that the log processing system cannot obtain data, and achieve the effect of improving the user experience

Pending Publication Date: 2021-12-07
BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The embodiment of the present invention provides a log processing method, system, device, client and storage medium, which solves the problem that the log processing system cannot obtain the data before the change operation

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log processing method, system and device, client and storage medium
  • Log processing method, system and device, client and storage medium
  • Log processing method, system and device, client and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0027] figure 1 It is a flow chart of a log processing method applicable to a log processing client provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where the obtained change operation log is processed to obtain data before and after the change operation respectively. The method can be executed by the log processing device provided in the embodiment of the present invention, and the device can be implemented by software and / or hardware.

[0028] see figure 1 , the method of the embodiment of the present invention specifically includes the following steps:

[0029] S110. Obtain a change operation log based on the middleware, wherein the change operation log is a corresponding log sent by the log acquisition server when the log is changed based on the middleware listening to the preset binary log of the relational database.

[0030] Among them, a relational database is a database based on a relational model, such as MySQL, SQLServer...

Embodiment 2

[0046] Before introducing the second embodiment, the ELK architecture in the prior art is analyzed in detail again: first, Logstash consumes high resources due to high CPU and memory usage during operation; in addition, it reads from the MySQL table Fetching log data, this reading process will put a lot of pressure on MySQL, and the reading result is only the updated data (that is, incremental update), and it cannot directly operate on the reading result. Secondly, Elasticsearch can provide the three functions of collecting, analyzing and storing log data. It is more suitable for application scenarios with more read operations, while most log processing systems involve write operations and read operations. Mostly it only appears in a small number of scenarios such as troubleshooting. Again, Kibana's visualization function is even more irrelevant in a log processing system that mainly records log data.

[0047] corresponding, figure 2 It is a schematic structural diagram of ...

Embodiment 3

[0052] image 3 It is a schematic structural diagram of a log processing system provided in Embodiment 3 of the present invention. This embodiment is optimized on the basis of the above-mentioned technical solutions. In this embodiment, optionally, the above-mentioned log processing system may further include a log search server, configured to search the log database according to the received search criteria input by the user, and match the searched ones with the search criteria The target data is returned. Wherein, explanations of terms that are the same as or corresponding to the above embodiments are not repeated here.

[0053] see image 3 , the log processing system of this embodiment may include: a log acquisition server 30 configured with middleware, a log processing client 31 configured with middleware, a log database 32, and a log search server 33, wherein the log acquisition service The terminal 30 is used to obtain the corresponding change operation log when mon...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a log processing method, system and device, a client and a storage medium. The method is applied to a log processing client side and comprises steps that a change operation log is obtained based on middleware, and the change operation log is a corresponding log sent when a log obtaining server side monitors that a binary log of a preset relational database is subjected to log change based on the middleware; according to the type of each message body in the change operation log, target message bodies belonging to the same transaction are screened out from the change operation log, the target message bodies are processed, a processing result is stored in a log database, and the processing result comprises data before the change operation and data after the change operation. According to the method, the effect of simultaneously obtaining the data before the change operation and the data after the change operation is achieved.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of computer applications, and in particular to a log processing method, system, device, client and storage medium. Background technique [0002] At present, most major systems use MySQL to manage data. For example, MySQL corresponding to the product system stores hundreds of millions of data, which covers product information, category information, and so on. Faced with such a large amount of data, in order to clearly record the user's operation process on these data, the product log processing system came into being. [0003] The existing commodity log system architecture is ELK (Elasticsearch+Logstash+Kibana) architecture, in which Logstash reads log data from MySQL as a pipeline and stores the read results in Elasticsearch, and Kibana is a web-based graphical interface , which visualizes log data stored in Elasticsearch. [0004] In the process of realizing the present invention...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/18G06F16/17G06F16/14G06F16/28
CPCG06F16/1815G06F16/1734G06F16/148G06F16/284
Inventor 江家宾李井新朱强
Owner BEIJING WODONG TIANJUN INFORMATION TECH CO LTD