Log processing method and device, equipment and storage medium

Abstract

The application provides a log processing method, device and equipment and a storage medium. The method comprises the following steps: obtaining a query request of a user, determining a target log from logs obtained from n business systems according to the query request, wherein n is a positive integer greater than 1, the query request comprises a request identification (ID) or a query word, the request ID is generated by a first business system in the n business systems when the logs of the n business systems are obtained, the query word comprises a keyword and / or a regular expression, and the target log is pushed. The required monitoring or analysis log is obtained from the logs of multiple software systems, monitoring or analysis is performed on the logs of multiple business systems, and accurate identification of abnormalities is possible.

Description

TECHNICAL FIELD

[0001] The present application relates to the computer technical field, and particularly relates to a log processing method and device, equipment and a storage medium. BACKGROUND

[0002] In the computer technical field, a log is used for monitoring or analyzing the running state of a software system, as a record of the running data of the software system running in a computer. For example, when the running of the software system is abnormal, the log of the software system can be used for abnormal analysis.

[0003] At present, because the logs of different software systems are isolated from each other, when multiple related software systems are analyzed, the logs of the multiple software systems cannot be obtained, and thus the abnormal analysis cannot be performed based on the logs of the multiple software systems. The abnormal analysis is performed on each software system based on the log of the software system, which results in low processing efficiency and even cannot accurately identify the abnormality. Therefore, how to query the required monitoring or analysis log based on the logs of multiple business systems is a problem to be solved. SUMMARY

[0004] The present application provides a log processing method and device, equipment and a storage medium, which can obtain the required monitoring or analysis log from the logs of multiple software systems, and thus can monitor the logs of the multiple software systems and provide the possibility of jointly performing abnormal analysis based on the logs of the multiple software systems.

[0005] In a first aspect, an embodiment of the present application provides a log processing method, which includes: obtaining a query request of a user; determining a target log from logs obtained from n business systems according to the query request, n is a positive integer greater than 1, the query request includes a request identification (ID) or a query word, the request ID is generated by a first business system in the n business systems when the logs of the n business systems are obtained, and the query word includes a keyword and / or a regular expression; and pushing the target log.

[0006] In a possible implementation, when the query request includes a request ID, the determining of the target log from the logs of the n business systems according to the query request includes: determining the target log as the logs of the n business systems according to the request ID.

[0007] In a possible implementation, the query request includes a keyword and / or a regular expression, and the determining of the target log from the logs of the n business systems according to the query request includes: determining the target log corresponding to the keyword and / or the regular expression from the logs of the n business systems according to the keyword and / or the regular expression.

[0008] In a possible implementation, the method further includes: receiving a plurality of first logs sent by the n service systems, the first logs including a request ID, a service ID, and log content, the request ID being generated by a first service system in the n service systems when the log of the n service systems is requested, and the service ID being an identification of a service process to which the log content corresponds; and screening the at least one first log according to the request ID and the receiving time of each first log in the at least one first log, to obtain the log of the n service systems.

[0009] In a possible implementation, the receiving the at least one first log sent by the n service systems includes: storing a plurality of first logs sent by the n service systems according to a first sorting rule by using a producer-consumer model, the first sorting rule being preset or indicated by a user instruction.

[0010] In a possible implementation, the receiving the plurality of first logs sent by the n service systems includes: dividing, by a producer thread in the producer-consumer model, the plurality of first logs sent by the n service systems to a log queue corresponding to the service system; and reading, by a consumer thread in the producer-consumer model, an offset of each of the plurality of first logs from a memory space, and extracting the plurality of first logs from the log queue corresponding to the n service systems respectively according to the offset of each of the plurality of first logs.

[0011] In a possible implementation, the method further includes: deleting a log in the log of the n service systems, which is generated for a time longer than a first time length, the first time length being a preset time length or being determined based on a query per second rate (QPS) of the n service systems.

[0012] In a second aspect, an embodiment of the present application provides a log processing method, which includes: a first service system generating a request ID, the request ID being used to request to obtain logs of the first service system and at least one second service system; the first service system sending the request ID to at least one second service system; the first service system generating at least one first log according to the request ID; and the first service system sending the at least one first log to a log monitoring system.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising: an acquisition unit configured to acquire a query request of a user; a processing unit configured to determine a target log from logs acquired from n service systems according to the query request, n being a positive integer greater than 1, the query request comprising a request identification (ID) or a query word, the request ID being generated by a first service system in the n service systems when the logs of the n service systems are acquired, and the query word comprising a keyword and / or a regular expression; and a pushing unit configured to push the target log.

[0014] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor; the memory stores computer-executed instructions; and the processor executes the computer-executed instructions stored in the memory, so that the processor executes the method in the first aspect and each implementation manner of the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor; the memory stores computer-executed instructions; and the processor executes the computer-executed instructions stored in the memory, so that the processor executes the method in the second aspect and each implementation manner of the second aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a storage medium, comprising: a readable storage medium and a computer program, the computer program being configured to implement the method in the first aspect and each implementation manner of the first aspect.

[0017] In a seventh aspect, an embodiment of the present application provides a storage medium, comprising: a readable storage medium and a computer program, the computer program being configured to implement the method in the second aspect and each implementation manner of the second aspect.

[0018] The log processing method, device, electronic device and storage medium provided by the embodiments of the present application can determine the target log from the logs of multiple service systems, and provide a possibility for accurately identifying an exception for monitoring or analyzing the logs of multiple service systems. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without any creative labor.

[0020] Figure 1 An application scenario diagram of a log processing system 100 provided by an embodiment of the present application is shown in the following figure:

[0021] Figure 2 A flowchart of a log processing method 200 provided for an embodiment of the present application is shown in FIG. 2;

[0022] Figure 3 An interaction flowchart of a log processing method 300 provided for an embodiment of the present application is shown in FIG. 3;

[0023] Figure 4 A structural diagram of a log monitoring system provided for an embodiment of the present application is shown in FIG. 4;

[0024] Figure 5 A structural diagram of a log monitoring system based on Kafka provided for an embodiment of the present application is shown in FIG. 5;

[0025] Figure 6 A structural diagram of an electronic device 400 provided for an embodiment of the present application is shown in FIG. 6;

[0026] Figure 7 A structural diagram of an electronic device 500 provided for an embodiment of the present application is shown in FIG. 7;

[0027] Figure 8 A structural block diagram of an electronic device 600 provided for an embodiment of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and superiorities of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0029] By monitoring the logs of the business system through the log monitoring platform, it is beneficial to discover and solve system problems in time. Since there are currently scenarios of multiple business systems cooperating, for example, an e-commerce platform can be composed of a commodity system, an order system, a stock system, an after-sales system, etc. Each business system maintains its own log data, for example, each business system corresponds to a log monitoring platform, the log of the business system is monitored through the log monitoring platform of each business system, and log analysis is performed. However, because the logs of each business system are isolated from each other, the logs of the business systems are monitored and analyzed one by one, which leads to a low processing effect of the logs, and the abnormal problem analysis is performed according to the logs of each business system one by one, which cannot jointly analyze the logs of multiple business systems comprehensively, leading to a low accuracy of the analysis result. Therefore, how to query the logs required to be monitored or analyzed based on the logs of multiple business systems is a problem to be solved at present.

[0030] To solve the above problems, the embodiment of the present application can determine the target log from the logs of multiple business systems, which provides the possibility of accurately identifying abnormalities for monitoring or analyzing the logs of multiple business systems.

[0031] Figure 1 An application scenario diagram of a log processing system 100 provided by the embodiment of the present application is shown. The log processing system 100 can include a log monitoring system 110 and n business systems 120, where n is a positive integer greater than 1. As shown in Figure 1 In some embodiments, the log monitoring system 110 and each business system 120 are deployed on different electronic devices, which can be servers, terminal devices, chips, etc., and the terminal devices can be smart phones, notebook computers, tablet computers, personal desktop computers, etc.

[0032] Optionally, the log monitoring system 110 and the n business systems 120 are deployed on one electronic device, or the log monitoring system 110 and the n business systems 120 are deployed on the same electronic device, or the log monitoring system 110 is deployed on a first electronic device and the n business systems 120 are deployed on a second electronic device, or the log monitoring system 110 is deployed on a first electronic device, an upstream business system in the n business systems is deployed on a second electronic device, and the remaining business systems are deployed on a third electronic device. The embodiment of the present application is not limited to the deployment modes exemplified above, and the log monitoring system and the n business systems can be deployed in any combination in at least one electronic device.

[0033] In some embodiments, the log monitoring system or the business system can be deployed in multiple electronic devices, and the operation of the system is realized by the cooperation of multiple electronic devices.

[0034] Optionally, the electronic device running the log monitoring system can be an ElasticSearch (ES) server, or the log monitoring system and the ES are connected through a wired or wireless manner.

[0035] Figure 2 A flow diagram of a log processing method 200 provided by the embodiment of the present application is shown. Figure 2 The method provided by the embodiment shown can be executed by the log monitoring system 110 or the electronic device on which the log monitoring system 110 is deployed. For example, Figure 1 The method includes:

[0036] S210: Obtain a query request of a user;

[0037] S220: According to the query request, determine a target log from the logs obtained from the n business systems, where n is a positive integer greater than 1.

[0038] S230: push the target log.

[0039] It is to be noted that the query request can be received from user input or from a user-side client.

[0040] In some embodiments, the log monitoring system can push a web visualization page to the user-side client, and the user can input the query instruction in the web visualization page. Optionally, the web visualization page is generated based on the ELK framework.

[0041] It is to be noted that the log monitoring system can provide logs obtained from the n business systems for the user to query the target log.

[0042] For example, the log monitoring system can obtain logs from the n business systems in advance and store them in the storage space of the log monitoring system or the storage space of the connected server. When the user queries the log, the target log is read.

[0043] It is to be noted that the logs of the n business systems can be full logs of the n business systems (i.e., all historical logs of the n business systems) or can be incremental logs of the n business systems (i.e., logs obtained between the current log obtaining time and the last log obtaining time).

[0044] Further, the log monitoring system pushes the target log to the user or the client of the user. For example, the target log can be displayed through a display device, which can be a display screen of an electronic device in which the log monitoring system is deployed, or a display device connected to the electronic device.

[0045] In the embodiments of the present application, the log monitoring system can provide at least two possible query modes.

[0046] Mode one: the log monitoring system can push the logs of the n business systems to the user as the target log.

[0047] In the mode one, the query instruction of the user obtained in the S210 should include a request identity (ID). It should be noted that the request ID is generated by a first business system in the n business systems when the logs of the n business systems are obtained. The first business system can be any business system in the n business systems, or can be a pre-specified business system in the n business systems. For example, the n business systems are business systems having an upstream and downstream relationship, and the most upstream business system is taken as the first business system. It can be understood that the upstream business system can pass instruction information or business data to the downstream business system, and the downstream business system can perform corresponding operations based on the instruction information or the business data, for example, after a product order is generated in the upstream business system, the downstream business system completes the payment of the product order.

[0048] It should be noted that the logs of the n business systems all have a mapping relationship with the request ID. For example, the logs are stored in the form of key-value pairs, that is, the request ID is the key and each log content is the value.

[0049] Mode two: the log monitoring system can determine the target log required by the query request in the logs of the n business systems.

[0050] In the mode two, the query instruction of the user obtained in the S210 should include a query word, which can include a keyword and / or a regular expression, for example. The log monitoring system matches the keyword and / or the regular expression in the logs of the n business systems to obtain the corresponding target log. Optionally, the keyword and / or the regular expression can be used to express the generation time, the storage time, the business ID, part or all of the characters of the log content of the required target log. The business ID is the business process identity corresponding to the log content, for example, when a business system executes a product sales process, a business ID is generated, and the logs generated when the business system executes the product sales process correspond to the business ID.

[0051] In the embodiments of the present application, by determining the required target log from the logs of the multiple business systems, it is possible to monitor or analyze the logs of the multiple business systems and to accurately identify the abnormality.

[0052] Figure 3 An interaction flow diagram of a log processing method 300 provided in the embodiments of the present application is shown in FIG. 3. As shown in FIG. 3, the method 300 takes the interaction between the log monitoring system and the business system as an example to illustrate the process of obtaining the logs of the n business systems by the log monitoring system. The log monitoring system can be the log monitoring system described above. Figure 3 Figure 1 ​The business system can be the business system 120 or an electronic device deployed with the business system. The method comprises at least part of the following steps:

[0053] S301: The first business system generates a request ID, which is used to request to obtain logs of the first business system and at least one second business system;

[0054] S302: The first business system sends the request ID to the at least one second business system;

[0055] S303: The first business system generates at least one first log according to the request ID;

[0056] S304: The first business system sends the at least one first log to the log monitoring system;

[0057] S305: Each of the at least one second business system generates at least one first log corresponding to the second business system according to the request ID;

[0058] S306: Each of the at least one second business system sends the at least one first log generated by the second business system to the log monitoring system.

[0059] Corresponding to S304 and S306, the log monitoring system receives a plurality of first logs sent by the first business system and the at least one second business system, the first log comprising a request ID, a business ID and log content, the request ID being generated by the first business system in the n business systems when obtaining logs of the n business systems (including the first business system and the at least one second business system), and the business ID being a business process identifier corresponding to the log content;

[0060] S307: The log monitoring system screens the at least one first log according to the request ID and the receiving time of each of the at least one first log, and obtains logs of the n business systems.

[0061] It should be noted that the execution order of S302 and S303 in the above embodiment is not limited. If S302 is executed before S303, the execution order of S303-S304 and S305-S306 in the embodiment is not limited.

[0062] It should be noted that the first business system can generate the request ID according to the user's log storage operation, or can generate the request ID at a preset time, or can generate the request ID periodically.

[0063] The request ID is a global ID, which can be generated by a universally unique identifier (UUID) or a snowflake algorithm, for example.

[0064] In S302, the first service system sends the request ID to at least one second service system. For example, the request ID can be sent to each second service system, or the request ID can be sent to a first layer of service systems downstream of the first service system, and the first layer of service systems sends the request ID to a second layer of service systems downstream of the first layer of service systems. For example, each service system transmits the request ID to the service system downstream, and it can be understood that pass-through means transparent transmission, that is, the request ID is directly transmitted from the receiving end to the target end without any change.

[0065] In S303, the first service system combines the request ID and the log content to obtain a first log. For example, the request ID is used as the key, and the log content is used as the value, and the obtained key-value pair is the first log.

[0066] In some embodiments, the first log further includes a service ID and / or a log generation time. The service ID and / or the log generation time exist when the log is generated, and can be understood as part of the log content.

[0067] S305 is similar to S303, which will not be described here.

[0068] In S304 or S306, each service system sends the first log to the log monitoring system, and the log monitoring system receives the first log sent by each service system. As a possible implementation, in order to facilitate monitoring of the logs of n service systems, the log monitoring system can store the multiple first logs sent by the n service systems according to a first sorting rule through a producer-consumer model of the log monitoring system, and the first sorting rule is preset or indicated by a user instruction.

[0069] For example, the log monitoring system can divide the multiple first logs sent by the n service systems into the log queues corresponding to the service systems through the producer thread in the producer-consumer model, and read the offsets of the multiple first logs from the memory space through the consumer thread in the producer-consumer model, and extract the multiple first logs from the log queues corresponding to the n service systems according to the offsets of the multiple first logs.

[0070] Optionally, the log monitoring system can be a system based on Kafka message queue, Kafka is a high-performance cross-language distributed publish / subscribe message queue.

[0071] The following is described by taking Kafka message queue as an example. Kafka puts the first log of a business system received by a producer thread into a topic corresponding to the business system. When the first log reaches the server (Broker) of Kafka, the log monitoring system starts multiple input / output (I / O) threads to consume the message (i.e., the first log). When Kafka consumes the message by a consumer thread, the consumer offset (i.e., the offset) is submitted to ZooKeeper, which is a distributed, open-source distributed application coordination service. The consumer offset can be read from ZooKeeper by the consumer, and the message is continued to be consumed based on the read consumer offset. However, the consumer offset is a variable that is updated very frequently, and frequent writing of the consumer offset to ZooKeeper reduces the performance of ZooKeeper. Therefore, in the embodiment, the offsets corresponding to the plurality of first logs can be determined in advance, and the offsets corresponding to the plurality of first logs are stored in the memory, and then the offset corresponding to each first log can be read from the content to extract the first log in the log queue. Thus, the frequent writing operation to ZooKeeper is avoided.

[0072] In some embodiments, since Kafka cannot guarantee the order and idempotency of messages, the first log extracted by the consumer thread can be put into a concurrent queue (sendQueue), and then a thread is used to take the message from the queue and put it into a Map structure cache, the key is the timestamp (timestap) of the message, i.e., the time when the log monitoring system receives the message, and the value is a priority queue. If the key already exists in the map, the latter directly covers the former, to achieve the purpose of deduplication.

[0073] Then a worker thread is used to take the message according to the user-defined sorting rule, and the message is stored in ES in order.

[0074] For example, Kafka stores the logs of n business systems in ES. Since the data volume of the stored logs is large, in order to optimize the performance of ES, the following optimizations are made in the embodiment: 1) the source function of the field that does not need to be queried is closed, and this is not stored in ES, to save disk space;

[0075] 2) Batch writing in multi-threaded mode, the number of writing threads is generally the same as the number of ES nodes, and the number of threads is determined by observing the performance curve of ES through Kibana (Kibana is an open source analysis and visualization platform) in the test environment; 3) Increase the refresh time of segments to obtain the latest log index; 4) In terms of memory allocation, reserve 50% of the memory of the ES system for the search engine (Lucene) to do file caching to support the search engine to undertake larger task quantity, and the memory of each ES node can be more than 64G; 5) Use SSD solid state disk.

[0076] In this embodiment, the log monitoring system receives a plurality of first logs sent by the n business systems, and obtains the logs of the n business systems by de-duplicating and sorting the plurality of first logs, and further stores the logs of the n business systems, so as to facilitate the user to monitor the logs of the n business systems, or query the target log in the logs of the n business systems, or perform log analysis based on the logs of the n business systems or the target log queried from the logs of the n business systems.

[0077] In some embodiments, the log monitoring system can also delete the logs in the logs of the n business systems that are generated for more than a first duration, which is a preset duration or determined based on the query rate per second (QPS) of the n business systems.

[0078] The first duration may, for example, be 7 days, for example, the log monitoring system can delete the logs of 7 days ago.

[0079] Or the first duration can be determined according to the QPS of the n business systems, it can be understood that the greater the QPS of a business system, the more logs the business system generates, and then the first duration should be shorter. For example, the first duration determined according to the QPS of the n business systems may, for example, be determined according to the maximum value in the QPS corresponding to each business system, or determined according to the minimum value in the QPS corresponding to each business system, or determined according to the median value in the QPS corresponding to each business system, or determined according to the average value in the QPS corresponding to each business system, etc.

[0080] Figure 4 A structural schematic diagram of a log monitoring system provided by the embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the log monitoring system comprises a log receiving module 101, a log processing module 102, a log storage module 103, and a log query module 104. Figure 4As shown, the log monitoring system includes at least one of the following modules: message sending module, message receiving module, message processing module, message storage module, message query module, and message deletion module. Specifically, the message sending module receives multiple first logs from n business systems via a producer thread and places them into a log queue; the message receiving module extracts the first logs placed in the log queue by the producer thread via a consumer thread; the message processing module deduplicates and sorts the extracted first logs; the message storage module stores the logs from the n business systems in Elasticsearch (ES) and improves storage performance by setting corresponding parameters and components of ES; the message query module determines the target log from the logs of the n business systems according to a query command; and the message deletion module automatically deletes expired logs from the logs of the n business systems to reduce storage pressure and improve log query speed.

[0081] Figure 5 This is a schematic diagram of the structure of a Kafka-based log monitoring system provided in an embodiment of this application. Figure 5 As shown, multiple producer threads in Kafka are used to receive the first logs sent by multiple business systems and put them into the Kafka broker. Then, consumer threads consume the messages and extract the first logs from the broker. In some embodiments, Kafka also includes ZooKeeper to provide consumer offsets for the consumer threads.

[0082] like Figure 5 As shown, the log monitoring system may also include a web visualization interface generated based on the ELK framework, used to receive user query commands.

[0083] like Figure 5 As shown, the log monitoring system can also periodically clean up expired logs through a log deletion thread.

[0084] Figure 6 This is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of this application, as shown below. Figure 6 As shown, the electronic device 400 includes:

[0085] Acquisition unit 410 is used to acquire user query requests;

[0086] Processing unit 420 is configured to determine the target log from the logs obtained from n business systems according to the query request, where n is a positive integer greater than 1. The query request includes a request identifier ID or a query term. The request ID is generated by the first business system among the n business systems when the logs of the n business systems are obtained. The query term includes keywords and / or regular expressions.

[0087] The pushing unit 430 is configured to push the target log.

[0088] The electronic device 400 provided in the embodiments of the present application comprises an obtaining module 410, a processing module 420 and a pushing module 430. The target log is determined from logs of a plurality of business systems, and then the logs of the plurality of business systems are monitored or analyzed. Compared with the prior art in which the logs of each business system are monitored and analyzed one by one, the processing efficiency is improved. In addition, the logs of the plurality of business systems are used for abnormal problem analysis, and the accuracy of the analysis result is improved.

[0089] In a possible design, the processing module 420 is specifically configured to determine, according to the request ID, that the target log is a log of the n business systems, and the query request comprises a request ID.

[0090] In a possible design, the processing module 420 is specifically configured to determine, according to the keyword and / or the regular expression, a target log corresponding to the keyword and / or the regular expression from the logs of the n business systems.

[0091] In a possible design, the electronic device 400 further comprises a receiving unit 440 configured to receive a plurality of first logs sent by the n business systems, wherein the first log comprises a request ID, a business ID and log content, the request ID is generated by a first business system in the n business systems when the log of the n business system is obtained, and the business ID is a business process identifier corresponding to the log content; and the processing unit is further configured to filter the at least one first log according to the request ID and the receiving time of each first log in the at least one first log, and obtain the log of the n business systems.

[0092] In a possible design, the receiving unit is specifically configured to store the plurality of first logs sent by the n business systems according to a first sorting rule through a producer-consumer model, and the first sorting rule is preset or indicated by a user instruction.

[0093] In a possible design, the receiving unit is specifically configured to divide, by a producer thread in the producer-consumer model, the plurality of first logs sent by the n business systems into the log queue corresponding to the business system.

[0094] The offset of each of the plurality of first logs is read from the memory space by a consumer thread in the producer-consumer model, and the plurality of first logs are extracted from the log queue corresponding to the n business systems respectively according to the offset of each of the first logs.

[0095] In one possible design, the processing unit 420 is further configured to: delete logs in the logs of the n business systems whose generation duration exceeds a first duration, wherein the first duration is a preset duration or is determined based on the query per second (QPS) of the n business systems.

[0096] The electronic device provided in the above embodiments can execute the technical solution of the log monitoring system side in the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0097] Figure 7 This is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of this application, as shown below. Figure 7 As shown, the electronic device 500 includes:

[0098] Processing unit 510 is used to generate a request ID, which is used to request the acquisition of logs from a first business system and at least one second business system;

[0099] Sending unit 520 is used to send the request ID to at least one second business system;

[0100] The processing unit 510 is further configured to generate at least one first log based on the request ID;

[0101] The sending unit 520 is also used to send the at least one first log to the log monitoring system.

[0102] The electronic device provided in the above embodiments can execute the technical solution of the first business system side in the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0103] This application also provides an electronic device, see [link to relevant documentation] Figure 8 The embodiments in this application are only used as examples. Figure 8 This is used as an example for illustration and does not mean that this application is limited to this.

[0104] Figure 8 This is a schematic structural diagram of an electronic device 600 provided in an embodiment of this application. Figure 6 The illustrated electronic device 600 includes a processor 610, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0105] Optionally, such as Figure 8 As shown, the electronic device 600 may further include a memory 630. The processor 610 can retrieve and run computer programs from the memory 630 to implement the methods described in the embodiments of this application.

[0106] The memory 630 can be a separate device independent of the processor 610, or can be integrated in the processor 610.

[0107] Optionally, as shown in the figure, the electronic device 600 can further include a transceiver 620, and the processor 610 can control the transceiver 620 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. Figure 8

[0108] The transceiver 620 can include a transmitter and a receiver. The transceiver 620 can further include an antenna, and the number of antennas can be one or more.

[0109] Optionally, the electronic device 600 can implement the corresponding processes of the log monitoring system in various methods of the embodiments of the present application, and details are not repeated here for the sake of brevity.

[0110] Optionally, the electronic device 600 can implement the corresponding processes of the business system in various methods of the embodiments of the present application, and details are not repeated here for the sake of brevity.

[0111] It should be understood that the processor of the embodiments of the present application can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method embodiments can be completed by integrated logic circuits or instruction forms of software in the processor. The processor mentioned above can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps of the above method.

[0112] ​It is to be understood that the memory in the embodiments of the present application can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM) or a flash memory. The volatile memory can be a random access memory (Random Access Memory, RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM). It should be noted that the memory of the system and method described herein is intended to include, but not limited to, these and any other suitable types of memory.

[0113] It should be understood that the above-mentioned memory is exemplary but not limiting, for example, the memory in the embodiments of the present application can also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM) and the like. That is, the memory in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memory.

[0114] The embodiment of the present application further provides a computer readable storage medium for storing the computer program.

[0115] Optionally, the computer readable storage medium can be applied to the electronic device in the embodiment of the present application, and the computer program makes the computer execute the corresponding process executed by the log monitoring system or the business system in the various methods of the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0116] The embodiment of the present application further provides a computer program product comprising computer program instructions.

[0117] Optionally, the computer program product can be applied to the electronic device in the embodiment of the present application, and the computer program instructions make the computer execute the corresponding process executed by the log monitoring system or the business system in the various methods of the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0118] The embodiment of the present application further provides a computer program.

[0119] Optionally, the computer program can be applied to the electronic device in the embodiment of the present application, and when the computer program runs on the computer, the computer executes the corresponding process executed by the log monitoring system or the business system in the various methods of the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0120] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware or in combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-mentioned devices, apparatuses and units can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0122] In several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the embodiments of the apparatus described above are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0123] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0124] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can be physically present separately, or two or more units can be integrated into one unit.

[0125] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. For such understanding, the technical solutions of the present application or the essential part or part of the technical solutions that make contributions to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0126] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A log processing method, characterized in that, The method is applied to a log monitoring system, and the method includes: Get the user's query request; According to the query request, the target log is determined from the logs obtained from n business systems, where n is a positive integer greater than 1. The query request includes a request identifier ID or query terms. The request ID is generated by the first business system among the n business systems when obtaining the logs, and is passed through the first business system to other business systems so that each business system generates its own first log based on the request ID. The logs obtained from the n business systems are determined based on the first logs corresponding to each business system. The query terms include keywords and / or regular expressions. Push the target log; The method further includes: Receive multiple first logs sent by the n business systems. The first log includes a request ID, a business ID, and log content. The business ID is the business process identifier corresponding to the log content. Based on the request ID and receiving time of each of the plurality of first logs, the plurality of first logs are filtered to obtain the logs of the n business systems; The receipt of multiple first logs sent by the n business systems includes: Using the producer thread in the producer-consumer model, the multiple first logs sent by the n business systems are divided into the log queues corresponding to the business systems. The consumer thread in the producer-consumer model reads the offsets of the multiple first logs from the memory space, and extracts the multiple first logs from the log queues corresponding to the n business systems according to the offsets of the first logs.

2. The method according to claim 1, characterized in that, When the query request includes a request ID, determining the target log from the logs of the n business systems based on the query request includes: Based on the request ID, the target log is determined to be the log of the n business systems.

3. The method according to claim 1, characterized in that, When the query request includes keywords and / or regular expressions, determining the target log from the logs of the n business systems based on the query request includes: Based on the keywords and / or regular expressions, determine the target logs corresponding to the keywords and / or regular expressions from the logs of the n business systems.

4. The method according to claim 1, characterized in that, Receiving at least one first log sent by the n business systems includes: The producer-consumer model is used to store multiple first logs sent by the n business systems according to a first sorting rule, which is either preset or indicated by user instructions.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Delete logs from the n business systems whose generation time exceeds a first duration, where the first duration is a preset duration or determined based on the query per second (QPS) of the n business systems.

6. A log processing method, characterized in that, The method includes: The first business system generates a request ID, which is used to request logs from the first business system and at least one second business system. The first business system sends the request ID to at least one second business system; The first business system generates at least one first log based on the request ID; The first business system sends at least one first log to the log monitoring system, so that the log monitoring system, through the producer thread in the producer-consumer model, divides the multiple first logs sent by the n business systems into the log queues corresponding to the business systems. Through the consumer thread in the producer-consumer model, it reads the offsets of the multiple first logs from the memory space and extracts the multiple first logs from the log queues corresponding to the n business systems according to their respective offsets. Based on the request ID and reception time of each first log, the system filters the multiple first logs to obtain the logs of the n business systems. When the log monitoring system receives a user's query request, it determines the target log from the logs obtained from the n business systems, where n is a positive integer greater than 1. The query request includes a request identifier ID or query terms, and the logs obtained from the n business systems are determined based on the first logs corresponding to each business system. The system then pushes the target log.

7. An electronic device, characterized in that, include: The retrieval unit is used to retrieve the user's query request; A processing unit is configured to determine a target log from logs obtained from n business systems according to the query request, where n is a positive integer greater than 1. The query request includes a request identifier ID or a query term, wherein the request ID is generated by the first business system among the n business systems when obtaining logs from the n business systems, and is transmitted to other business systems through the first business system, so that each business system generates a first log corresponding to its own business system according to the request ID. The logs obtained from the n business systems are determined based on the first logs corresponding to each business system. The query term includes keywords and / or regular expressions. The push unit is used to push the target log; The receiving unit is used to receive multiple first logs sent by the n business systems. The first log includes a request ID, a business ID, and log content. The business ID is a business process identifier corresponding to the log content. The processing unit is further configured to filter the plurality of first logs according to the request ID and receiving time of each first log in the plurality of first logs to obtain the logs of the n business systems; The receiving unit is specifically configured to, through the producer thread in the producer-consumer model, divide the multiple first logs sent by the n business systems into the log queues corresponding to the business systems; through the consumer thread in the producer-consumer model, read the offsets of the multiple first logs from the memory space, and extract the multiple first logs from the log queues corresponding to the n business systems respectively according to the offsets of the first logs.

8. An electronic device, characterized in that, include: A processing unit is configured to generate a request ID, which is used to request logs from a first business system and at least one second business system. A sending unit is used to send the request ID to at least one second business system; The processing unit is also configured to generate at least one first log based on the request ID; The sending unit is further configured to send the at least one first log to the log monitoring system, so that the log monitoring system, through the producer thread in the producer-consumer model, divides the multiple first logs sent by the n business systems into the log queues corresponding to the business systems, and through the consumer thread in the producer-consumer model, reads the offsets of the multiple first logs from the memory space, and extracts the multiple first logs from the log queues corresponding to the n business systems according to the offsets of the first logs; filters the multiple first logs according to the request ID and reception time of each first log to obtain the logs of the n business systems; when the log monitoring system obtains a user's query request, it determines the target log from the logs obtained from the n business systems according to the query request, where n is a positive integer greater than 1, the query request includes a request identifier ID or query terms, and the logs obtained from the n business systems are determined according to the first logs corresponding to each business system; Push the target log.

9. An electronic device, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

Images