Real-time monitoring method, system and equipment for tracking log and storage medium

Abstract

The invention relates to / belongs to the technical field of data processing, and provides a real-time monitoring method, system and equipment for tracking logs and a storage medium. The technical problem that in the prior art, an existing software monitoring system cannot provide a real-time running condition view is solved. Comprising the following steps: acquiring a tracking band log data set of a service system, and analyzing the tracking band log data set to obtain an analyzed data set; acquiring a request of each piece of analysis data in the analysis data set, performing pairing association on the analysis data with the same request to obtain pairing data, and performing loopback calculation on the pairing data to obtain a loopback message message set; calculating an index of each loopback message in the loopback message set, and performing classification aggregation calculation on the loopback message set based on the indexes to obtain a plurality of pieces of aggregation index data; and pushing the aggregated index data to a front-end information display module in real time for real-time monitoring.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a real-time monitoring method, system, device and storage medium for tracking band logs. Background Art

[0002] Tracking log is a specific type of log that usually contains information about all the key steps of a transaction or request from the beginning to the end. In the field of civil aviation, this log is crucial to ensure the accurate delivery of flight information, process passenger data, maintain system security, etc.

[0003] In the prior art, each step in the business process can be recorded in detail by integrating tracking and logging. At the same time, based on the tracking and logging, the software monitoring system continuously monitors the operation status, performance indicators, abnormal behaviors and key business processes of the business system.

[0004] The prior art has the following technical problems:

[0005] 1. With the rapid development of business needs and the accelerated changes in software conditions, the existing software monitoring system cannot provide a real-time view of the operation status.

[0006] 2. There is a significant delay in the chain from the generation of tracking logs to the final display, which affects the overall response speed.

[0007] 3. In the existing technology, some links do not meet the real-time requirements, thus affecting the real-time efficiency of the entire process. Summary of the invention

[0008] The present invention provides a real-time monitoring method, system, electronic device and storage medium for tracking logs, aiming to solve the technical problems existing in the above-mentioned prior art that the existing software monitoring system cannot provide a real-time operation status view; there is a significant delay in the chain from the generation of tracking logs to the final display, which affects the overall response speed; and some links do not meet the real-time requirements, thereby affecting the real-time efficiency of the entire process.

[0009] The technical solution of the present invention to solve the above technical problem is as follows: a real-time monitoring method for tracking a log, comprising:

[0010] Acquire a tracking tape log data set of the business system, and parse the tracking tape log data set to obtain a parsed data set;

[0011] Obtaining requests for each parsed data in the parsed data set, pairing and associating parsed data with the same request to obtain paired data, performing loopback calculation on the paired data to obtain a loopback message set;

[0012] Calculating the index of each loopback message in the loopback message set, and performing classified aggregation calculation on the loopback message set based on the index to obtain a plurality of aggregated index data;

[0013] The aggregated indicator data is pushed to the front-end information display module in real time for real-time monitoring.

[0014] Furthermore, the above-mentioned acquisition of the tracking band log data set of the business system and parsing the tracking band log data set to obtain the parsed data set specifically includes:

[0015] The tracking band log data in the tracking band log data set is received using a streaming computing method, and each tracking band log data is processed separately, and the content of each tracking band log data is parsed according to the tracking band log specification to obtain parsed data, wherein the parsed data includes log time information, software ownership information and business description information.

[0016] Further, the loopback message set obtained specifically includes:

[0017] Obtain a request ID for each piece of parsed data in the parsed data set, pair all parsed data with the same request ID to obtain paired data, calculate request attribute values ​​of the paired data, and obtain a loopback message; wherein the loopback message set is a set of all obtained loopback messages.

[0018] Furthermore, the above-mentioned multiple aggregated indicator data specifically include:

[0019] Configure the window time of the time window, aggregate and calculate all loopback messages within the time window, and obtain aggregate indicator data, which includes: average response time, request volume, error request volume and timeout request volume.

[0020] Further, the above-mentioned real-time pushing of the aggregated indicator data to the front-end information display module for real-time monitoring includes:

[0021] Pushing the aggregated indicator data to the front-end information display module in real time via the WebSocket protocol;

[0022] While pushing the aggregated indicator data to the front-end information display module in real time, the aggregated indicator data is stored synchronously.

[0023] Furthermore, the above also includes:

[0024] Through HTTP requests, the stored aggregate indicator data is queried and extracted, and the extracted aggregate indicator data is visualized.

[0025] Furthermore, the above-mentioned visualization of the extracted aggregated indicator data through the front-end information display module specifically includes:

[0026] The aggregated indicator data is obtained, the aggregated indicator data is visualized to obtain visualization data, and the visualization data is displayed; wherein the visualization data includes chart data and text data.

[0027] In a second aspect, in order to solve the above technical problems, the present invention also provides a real-time monitoring system with tracking logs, comprising:

[0028] A log message parsing module, used to obtain a tracking band log data set of a business system, and parse the tracking band log data set to obtain a parsed data set;

[0029] A log message loopback module, used for obtaining requests for each parsed data in the parsed data set, pairing and associating parsed data with the same request to obtain paired data, performing loopback calculation on the paired data to obtain a loopback message set;

[0030] An information aggregation module, used to calculate the index of each loopback message in the loopback message set, and perform classified aggregation calculation on the loopback message set based on the index to obtain multiple aggregation index data;

[0031] The information communication module is used to push the aggregated indicator data to the front-end information display module in real time for real-time monitoring.

[0032] In the third aspect, in order to solve the above-mentioned technical problem, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the real-time monitoring method with tracking log of the present application is implemented.

[0033] In a fourth aspect, in order to solve the above-mentioned technical problem, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the real-time monitoring method of tracking with logs of the present application is implemented.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] 1. The present invention adopts the log specification of the Air Traffic Information Tracking Band to ensure that the log format of all business systems is consistent, which is convenient for parsing and processing. The stable log input source reduces data processing errors caused by inconsistent log formats or poor quality, and improves the stability and reliability of the overall system.

[0036] 2. The present invention processes large-scale real-time data through a streaming computing framework, which can support 60,000 transactions per second (TPS) and meet high concurrency requirements.

[0037] 3. The present invention uses WebSocket to push real-time data, ensuring that the front-end display module can update the latest indicators in real time and provide a seamless user experience. It also combines the HTTP protocol to implement historical data query functions, which not only ensures the rapid transmission of real-time data, but also supports complex historical data analysis, meeting the demands of more practical scenarios.

[0038] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic flow chart of a real-time monitoring method for tracking band logs according to an embodiment of the present invention is shown;

[0041] Figure 2 A schematic structural diagram of a real-time monitoring system for tracking band logs according to an embodiment of the present invention is shown;

[0042] Figure 3 A schematic structural diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Figure 1 A flow chart of a real-time monitoring method for tracking band logs according to an embodiment of the present invention is shown. Figure 1 As shown, a real-time monitoring method for tracking with logs according to an embodiment of the present invention includes:

[0045] Acquire a tracking tape log data set of the business system, and parse the tracking tape log data set to obtain a parsed data set;

[0046] Obtaining requests for each parsed data in the parsed data set, pairing and associating parsed data with the same request to obtain paired data, performing loopback calculation on the paired data to obtain a loopback message set;

[0047] In this embodiment, pairing and association are performed on the input, intermediate processing, output and other log information generated during a business request process to calculate business attributes such as request response time.

[0048] Calculating the index of each loopback message in the loopback message set, and performing classified aggregation calculation on the loopback message set based on the index to obtain a plurality of aggregated index data;

[0049] Pushing the aggregated index data to the front-end information display module in real time;

[0050] Optionally, it also includes: using the front-end information display module to visualize the aggregated indicator data in real time.

[0051] In this embodiment, the tracking band log data is the log information of the instantaneous operation record, which mainly includes three parts: the message header, the extension area and the content area, and each part is separated by a line break. The types of tracking band log data include input logs, output logs and process logs. Each complete request process must include input logs and output logs. The index is used to classify multiple instantaneous tracking band logs of the same request, and then calculate and count them. The final aggregated index data includes: request volume, timeout request volume, error request volume, average response time, etc.

[0052] Optionally, obtaining a tracking log data set of a business system, and parsing the tracking log data set to obtain the parsed data set specifically includes:

[0053] The tracking band log data in the tracking band log data set is received using a streaming computing method, and each tracking band log data is processed separately, and the content of each tracking band log data is parsed according to the tracking band log specification to obtain parsed data, wherein the parsed data includes log time information, software ownership information and business description information.

[0054] In this embodiment, the parsed data is verified after parsing. If the verification fails, it is considered to be abnormal data and filtered out without entering the subsequent processing link. Specifically:

[0055] 1. Use Spark Structured Streaming to quickly and efficiently obtain the original tracking log messages from the log source;

[0056] 2. Split each original tracking log message into three parts: message header, extension area and content area.

[0057] 3. Parse the message header and use regular expressions to match and extract key information according to the tracking log specification. Key information includes: log operation time accurate to milliseconds (yyyyMMddHHMMSS.sss); unique message identifier representing the specific request operation; inbound and outbound message identifier representing the log type, where DAUDIT_IN is a request message and DAUDIT_OUT is a response message.

[0058] 4. Parse the extended area, and extract the first message attribute for the statistical dimension according to the statistical requirements of the indicator, where the first message attribute includes: source address (tap-orisvc) and destination address (tap-svc);

[0059] 5. Parse the content area and extract the second message attributes used for the statistical dimension according to the statistical requirements of the indicator. The second message attributes include: service type (msgtype) and airline identifier (airline);

[0060] 6. After parsing, verify the tracking log messages and filter out illegal messages, such as messages whose unique identifiers cannot be parsed; filter out non-business related messages, such as process log messages.

[0061] 7. Send the parsed and verified tracking log message (parsed data) to the Kafka message queue.

[0062] Optionally, the loopback message set includes:

[0063] Obtain a request ID for each piece of parsed data in the parsed data set, pair all parsed data with the same request ID to obtain paired data, calculate request attribute values ​​of the paired data, and obtain a loopback message; wherein the loopback message set is a set of all obtained loopback messages.

[0064] In this embodiment, a batch timeout pairing scheme is used to process the parsed data set, specifically including:

[0065] 1. Obtain parsed data from the Kafka message queue, extract the request message and response message of the same request based on the message unique identifier as the group identifier. Then, based on the preset window time, treat the logs that fail to be successfully matched within the window time as timed out or unresponsive requests.

[0066] 2. Based on paired request and response messages, calculate basic indicators (such as request response time, i.e., the time difference between the log time of the response message and the log time of the request message), and define request attributes (such as timeout, error). Among them, a request with only a request message log but no response message log is defined as a timeout request; based on the defined business response time requirements, a message that exceeds the response time threshold is also defined as a timeout message (identified by the attribute isTimeOut=1 in the loopback message); in the response message log, requests with abnormal results are identified and defined as error requests (identified by the attribute isError=1 in the loopback message);

[0067] 3. Send the calculated loopback message information to the Kafka message queue.

[0068] In this embodiment, 200,000 logs are obtained for pairing each time, and the unpaired logs are stored in memory. The unpaired logs are eliminated after more than 2 minutes, so as to maintain a balance between high performance and high reliability.

[0069] Optionally, multiple aggregated indicator data are obtained, including:

[0070] Configure the window time of the time window, aggregate and calculate all loopback messages within the time window, and obtain aggregate indicator data, which includes: average response time, request volume, error request volume and timeout request volume.

[0071] In this embodiment, the Spark Structured Streaming streaming computing framework technology is used to configure a time window (1 minute) for index aggregation statistics, specifically:

[0072] 1. Based on the demand index calculation rules, filter out loopback message data that is not related to the index. For example: business logs of non-designated airlines

[0073] 2. Configure controllable window time and aggregate indicators according to different dimensions (such as airlines, channels, offices, etc.). Generate aggregate indicator data such as average response time, number of requests, number of error requests, and number of timed requests.

[0074] Optionally, pushing the aggregated indicator data to a front-end information display module in real time for real-time monitoring includes:

[0075] Pushing the aggregated indicator data to the front-end information display module in real time via the WebSocket protocol;

[0076] While pushing the aggregated indicator data to the front-end information display module in real time, the aggregated indicator data is stored synchronously.

[0077] Optionally, also include:

[0078] Through HTTP requests, the stored aggregate indicator data is queried and extracted, and the extracted aggregate indicator data is visualized.

[0079] In this embodiment, the required historical data is obtained from the information communication module through an HTTP request, and the chart is rendered.

[0080] Optionally, using the front-end information display module to visualize the aggregated indicator data in real time specifically includes:

[0081] The aggregated indicator data is obtained, the aggregated indicator data is visualized to obtain visualization data, and the visualization data is displayed; wherein the visualization data includes chart data and text data.

[0082] Based on Figure 1 The same principle as the method shown in , the embodiment of the present invention also provides a real-time monitoring system with tracking logs, such as Figure 2 As shown in, including:

[0083] A log message parsing module, used to obtain a tracking band log data set of a business system, and parse the tracking band log data set to obtain a parsed data set;

[0084] A log message loopback module, used for obtaining requests for each parsed data in the parsed data set, pairing and associating parsed data with the same request to obtain paired data, performing loopback calculation on the paired data to obtain a loopback message set;

[0085] An information aggregation module, used to calculate the index of each loopback message in the loopback message set, and perform classified aggregation calculation on the loopback message set based on the index to obtain multiple aggregation index data;

[0086] An information communication module, used to push the aggregated indicator data to the front-end information display module in real time;

[0087] Optionally, a front-end information display module is also included, which is used to use the front-end information display module to visualize the aggregated indicator data in real time.

[0088] Optionally, the information communication module includes:

[0089] A real-time data push unit, connected to the front-end information display module, for pushing the aggregated indicator data to the front-end information display module in real time;

[0090] The historical data processing unit is connected to the front-end information display module, and is used to push the aggregated indicator data to the front-end information display module in real time through the WebSocket protocol, store the aggregated indicator data synchronously, and query and extract the required aggregated indicator data based on HTTP requests and upload it to the front-end information display module.

[0091] The real-time monitoring system with tracking logs of the embodiment of the present invention can execute the real-time monitoring method with tracking logs provided by the embodiment of the present invention, and the implementation principle is similar. The actions performed by each module and unit in the real-time monitoring system with tracking logs in each embodiment of the present invention correspond to the steps in the real-time monitoring method with tracking logs in each embodiment of the present invention. For the detailed functional description of each module of the real-time monitoring system with tracking logs, please refer to the description in the corresponding real-time monitoring method with tracking logs shown in the previous text, which will not be repeated here.

[0092] Among them, the above-mentioned real-time monitoring system with tracking logs can be a computer program (including program code) running in a computer device, for example, the real-time monitoring system with tracking logs is an application software; the application software can be used to execute the corresponding steps in the method provided in an embodiment of the present invention.

[0093] The modules involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a module does not limit the module itself in some cases.

[0094] Based on the same principle as the method shown in the embodiments of the present invention, an electronic device is also provided in the embodiments of the present invention, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the method shown in any embodiment of the present invention by calling the computer program.

[0095] In an alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3 The electronic device shown includes: a processor and a memory. The processor and the memory are connected, such as through a bus. Optionally, the electronic device may also include a transceiver, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver is not limited to one, and the structure of the electronic device does not constitute a limitation on the embodiments of the present invention.

[0096] The bus may include a path to transmit information between the above components. The bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0097] The memory can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disc, optical disk, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0098] The memory is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor. The processor is used to execute the application code stored in the memory to implement the content shown in the above method embodiment.

[0099] The electronic device may also be a terminal device. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0100] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0101] According to another aspect of the present invention, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the implementation modes of the above-mentioned various embodiments.

[0102] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0104] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

Claims

1. A real-time monitoring method with tracking logs, characterized in that: The method comprises: Acquire a tracking tape log data set of the business system, and parse the tracking tape log data set to obtain a parsed data set; Obtaining requests for each parsed data in the parsed data set, pairing and associating parsed data with the same request to obtain paired data, performing loopback calculation on the paired data to obtain a loopback message set; Calculating the index of each loopback message in the loopback message set, and performing classified aggregation calculation on the loopback message set based on the index to obtain a plurality of aggregated index data; The aggregated indicator data is pushed to the front-end information display module in real time for real-time monitoring.

2. A real-time monitoring method for tracking band logs according to claim 1, characterized in that: Acquiring a tracking log data set of a business system, and parsing the tracking log data set to obtain the parsed data set specifically includes: The tracking band log data in the tracking band log data set is received using a streaming computing method, and each tracking band log data is processed separately, and the content of each tracking band log data is parsed according to the tracking band log specification to obtain parsed data, wherein the parsed data includes log time information, software ownership information and business description information.

3. A real-time monitoring method for tracking band logs according to claim 1, characterized in that: The loopback message set includes: Obtain a request ID for each piece of parsed data in the parsed data set, pair all parsed data with the same request ID to obtain paired data, calculate request attribute values ​​of the paired data, and obtain a loopback message; wherein the loopback message set is a set of all obtained loopback messages.

4. A real-time monitoring method for tracking band logs according to claim 1, characterized in that: The following aggregated indicator data are obtained: Configure the window time of the time window, aggregate and calculate all loopback messages within the time window, and obtain aggregate indicator data, which includes: average response time, request volume, error request volume and timeout request volume.

5. A real-time monitoring method for tracking band logs according to claim 1, characterized in that: Pushing the aggregated indicator data to the front-end information display module in real time for real-time monitoring includes: Pushing the aggregated indicator data to the front-end information display module in real time via the WebSocket protocol; While pushing the aggregated indicator data to the front-end information display module in real time, the aggregated indicator data is stored synchronously.

6. A real-time monitoring method for tracking band logs according to claim 5, characterized in that: Also includes: Through HTTP requests, the stored aggregate indicator data is queried and extracted, and the extracted aggregate indicator data is visualized through the front-end information display module.

7. A real-time monitoring method for tracking band logs according to claim 6, characterized in that: The extracted aggregated indicator data is visualized through the front-end information display module, including: The aggregated indicator data is obtained, the aggregated indicator data is visualized to obtain visualization data, and the visualization data is displayed; wherein the visualization data includes chart data and text data.

8. A real-time monitoring system with tracking logs, characterized in that: include: A log message parsing module, used to obtain a tracking band log data set of a business system, and parse the tracking band log data set to obtain a parsed data set; A log message loopback module, used for obtaining requests for each parsed data in the parsed data set, pairing and associating parsed data with the same request to obtain paired data, performing loopback calculation on the paired data to obtain a loopback message set; An information aggregation module, used to calculate the index of each loopback message in the loopback message set, and perform classified aggregation calculation on the loopback message set based on the index to obtain multiple aggregation index data; The information communication module is used to push the aggregated indicator data to the front-end information display module in real time for real-time monitoring.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images