Log processing method and apparatus, electronic device and readable storage medium

Abstract

The present disclosure relates to a log processing method and apparatus, an electronic device and a readable storage medium. An airport device and a background system can be unified by means of trace log identifiers, log data generated by the system is aggregated on the basis of the trace log identifiers to obtain call chain data of each trace, and an explicit chain diagram is drawn for interface call relationships between applications in a departure control system, so that maintenance personnel can clearly know a call chain where the generated log data is located, and full-chain traceability of log data from an airport front-end device to the background system is realized. The maintenance personnel can conveniently and quickly locate a call chain where problematic log data is located, and automated testing of service calling can be assisted while the problem location difficulty is reduced.

Description

Log processing method, device, electronic device and readable storage medium Technical Field

[0001] The present disclosure relates to the field of data monitoring technology, and in particular to a log processing method, device, electronic device, and readable storage medium. Background Art

[0002] As the departure system becomes more open, the scale and traffic of the operational software increases, leading to an increasing volume of system logs. This volume of logs encompasses the entire process from front-end equipment to back-end operations. This makes troubleshooting airport system issues difficult for maintenance personnel, as they face such a flood of log data and are unable to fully trace the entire process from front-end equipment to back-end operations.

[0003] Public content

[0004] This disclosure section is provided to briefly introduce concepts that will be described in detail in the detailed description section below. This disclosure section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] The present disclosure provides a log processing method, device, electronic device and readable storage medium, which can realize the tracking of the link that generates log data, and effectively reduce the difficulty of troubleshooting and tracking airport system problems.

[0006] To achieve the above objectives, the present disclosure provides the following technical solutions:

[0007] A log processing method provided according to a specific embodiment of the present disclosure includes:

[0008] Obtain the original log data of each tracking band based on the message queue;

[0009] Filtering target log data containing corresponding tracking band log identification identifiers from each original log data;

[0010] Aggregating the target log data based on the tracking band log identification identifier of each target log data to obtain call link data;

[0011] After associating the obtained call link data with the corresponding tracking log identifier, it is stored in a preset relational database.

[0012] As can be seen from the above technical solutions, the present disclosure discloses a log processing method that can connect airport equipment and backend systems through tracking band log identifiers, aggregate the log data generated by the system according to the tracking band log identifiers, and obtain the call link data of each tracking band. This allows for a clear link diagram of the interface call relationship between various applications in the departure system, allowing maintenance personnel to clearly understand the call link where the generated log data is located, and achieve full-link tracking of log data from airport front-end equipment to backend systems. This facilitates maintenance personnel to quickly locate the call link where the problem log data is located, reducing the difficulty of problem location while assisting in the automated testing of business calls. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0014] FIG1 is a flow chart of a log processing method provided by an embodiment of the present disclosure;

[0015] FIG2 is a structural diagram of a log processing device provided by an embodiment of the present disclosure;

[0016] FIG3 is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0018] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0019] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0021] 1 , the present disclosure provides a log processing method, which may include the following steps:

[0022] 101. Obtain original log data of each tracking band based on the message queue.

[0023] Specifically, a tracking belt is a link from the airport's front-end equipment to the airport's back-end system that tracks each departure transaction. The collected raw log data can be transmitted to a Kafka queue (a high-throughput distributed publish-subscribe messaging system) as a data source.

[0024] 102. Target log data containing corresponding tracking band log identification identifiers are filtered out from each original log data.

[0025] Specifically, all received open software tracking tapes can be filtered based on the log identification identifier (globalid) of the departure tracking tape, and only the tracking tape log data containing the globalid is retained. The filtered log data will be stored in HDFS (a distributed file system) for use in problem troubleshooting.

[0026] 103. Based on the tracking band log identification identifier of each target log data, aggregate the target log data to obtain call link data.

[0027] The message header contains information such as software code, server name and transaction ID, and the extended area contains identification information such as the departure transaction global log identification identifier (Globalid), interface name (Action), caller (OriAppCode) and callee (DesAppCode) (PS: different protocol types have different methods and dependent fields for identifying the caller / callee, and the example uses Protocol = 5), etc. The missing caller information means that the message is transferred from the external system of Air China (recorded as exesys).

[0028] Application servers, as peripheral systems of large mainframe systems, contain all the input and output of the mainframe system. Based on the distinction between calls within the system and calls outside the system, logs are divided into DAUDIT and STAT types. DAUDIT types represent calls to modules within the system, while STAT types represent interactions between the system and other systems. Each log type includes IN and OUT directions. Logs are further divided into request and response types, represented by MsgType (values ​​are 1: synchronous request message, 2: asynchronous request message, 3: synchronous reply message, 5: asynchronous reply message, and 6: no reply message). Since inter-deployment calls are also recorded as STAT, analysis can only obtain STAT_IN logs with MsgType 1 / 2 / reuqest. The response time (the value of the timecost field) in STAT_OUT logs can then be used to determine the order of request calls. Because timecost represents the total time from request issuance to response, the software with the longest response time should be the one that initiated the request first. Similarly, each STAT_IN log contains information about the calling and called software and the called interface. Based on this information, we can derive outbound interface call relationship data for all logs with this globalID, such as "software 1: interface 1 -> software 2: interface 2 -> software 3: interface 3." A specific example is as follows:

[0029] If the original data obtained is (the interface name is in brackets):

[0030] exesys->A(A1);A->B(B1),A->B(B2);BC(C1), then the interface call relationship is recorded as:

[0031] It is understandable that those skilled in the art may adjust the formation of the call link according to the needs of actual applications, and no limitation is imposed here.

[0032] 104. After associating the obtained call link data with the corresponding tracking log identifier, the data is stored in a preset relational database.

[0033] Specifically, based on the HDFS data aggregated and saved by globalID, a call chain analysis is performed for each globalID. The analyzed chain data is stored in the EDB (Enterprise Database) database base table for later manual or automated batch verification. The query interface offers two main functionalities: business monitoring and departure interface call relationship analysis. The business monitoring interface provides pages such as the monitoring center, query center, software access volume, compliance query, and software non-compliance log statistics. The departure interface call relationship analysis page provides functions such as departure interface call relationship (design), departure interface call relationship (actual), departure interface call relationship (comparison), departure interface management, departure interface call relationship diagram verification and management, non-compliance data statistics & sub-node missing statistics, and email notifications. This log processing method addresses the lack of full-chain tracking, automated testing, and monitoring in the departure field, from airport front-end equipment to the back-end. A global log identifier allows for integrated querying of full-chain tracking logs between airport equipment and back-end systems, simplifying the troubleshooting and tracing of airport system issues.

[0034] In another specific embodiment of the present disclosure, in order to improve the reliability of call link determination, after filtering out target log data containing corresponding tracking band log identification identifiers from each original log data, each target log data can be subjected to compliance verification, and the target log data that has not passed the verification can be stored in a distributed file system for use in troubleshooting.

[0035] Specifically, when performing basic compliance checks on logs, you can filter out logs that meet the following four conditions:

[0036] 1) The globalid format is incorrect.

[0037] 2) Missing source or destination.

[0038] 3) Missing transaction type (MsgType) protocol=1 / 2 / 3 / 4 / 5.

[0039] 4) Protocol is missing.

[0040] The filtered logs are stored in HDFS for troubleshooting.

[0041] The verified target log data is stored in a distributed database, and the target log data in the distributed database is aggregated every first preset time period. The aggregated target log data is stored in a distributed file system based on the tracking band log identification identifier.

[0042] Specifically, the log data stored in hbase (a distributed database) is aggregated every 10 minutes according to the globalid using spark-streaming technology. The aggregated data is stored in HDFS, and different globalids are stored in files in HDFS according to the hash table. The data stored in HDFS can be processed so that the query center can query the logs based on the globalid and provide the interface call relationship of the globalid. It is provided to the automated test call in the form of an interface to achieve the goal of automated testing of all software for departure opening. The basis of automated testing is to have a designed interface call relationship diagram and an actual interface call relationship diagram. Then, by comparing the design and the actual call relationship diagram, it can be determined whether the actual call relationship diagram is consistent with the design. If it is consistent with the design, the test is considered to have passed, and if it is inconsistent with the design, the test is considered to have failed. For example, the following 9 interfaces can be provided:

[0043] Interface 1: Query the interface list of all software.

[0044] Interface 2: Query the link list of the interface (supports batch query).

[0045] Interface 3: Query link verification results.

[0046] Interface 4: Query the page link of the link map.

[0047] Interface 5: Returns the global design call relationship data of the specified interface.

[0048] Interface 6: Provides a query interface for actual call pairwise relationships. (By using Spark Streaming technology to perform offline link data analysis on all logs from the previous day, pairwise call relationships are generated and stored in the EDB database. This is then used for querying the departure interface call relationship comparison page, forming a comparison chart of the actual call links and the designed call links.)

[0049] Interface seven: Provide a link to a comparison query page.

[0050] Interface 8: Create a new page to implement the departure interface call relationship comparison page and compare the actual interface call relationship effect.

[0051] Interface 9: Provides automated testing to query smoke test coverage.

[0052] The function of batch automatic confirmation of the design interface call relationship is implemented by performing link serial analysis on each globalid based on the HDFS data saved after aggregation according to the globalid mentioned above. The analyzed link data is stored in the EDB database base table. At the same time, the link data in the base table will be synchronized at every hour. This synchronization operation will take all the link graph data of the globalid generated in the previous hour to the EDB to-be-confirmed database table to check whether the globalid with the same call link data already exists. If it does, it does not need to be synchronized to the to-be-confirmed database. If not, the link graph data of this globalid needs to be synchronized to the to-be-confirmed database for later manual confirmation or automatic batch confirmation.

[0053] Of course, in addition to using spark-streaming, you can also use flink for processing. flink can truly achieve the effect of real-time stream processing, while spark-streaming only simulates real-time stream processing. In the future, when the log volume reaches a certain amount and the data analysis performance cannot keep up, you can consider using flink for processing.

[0054] In some specific embodiments of the present disclosure, since globalid is a global business ID recorded when it is initiated from the airport front-end software, there will be relatively many transactions involved. Later, when it is transmitted to other software, a different transaction ID will be generated according to different transactions. Since the above analysis may generate a lot of incorrect interface call relationship data, the call link data can be adjusted and grouped based on the transaction identifier generated on the call link, and then relatively accurate departure interface call relationship data will be generated.

[0055] Although the operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.

[0056] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in a different order or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0057] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0058] 2 , an embodiment of the present disclosure further provides a log processing device, which can implement the various steps of the above-mentioned log processing method when running. The device may include:

[0059] The log data acquisition module 201 is used to acquire the original log data of each tracking band based on the message queue.

[0060] The log data screening module 202 is configured to screen out target log data containing corresponding tracking band log identification identifiers from each original log data.

[0061] The log data aggregation module 203 is used to aggregate the target log data based on the tracking log identification of each target log data to obtain call link data.

[0062] The call link generation module 204 is used to associate the obtained call link data with the corresponding tracking log identifier and store them in a preset relational database.

[0063] Furthermore, the log data screening module 202 is further configured to:

[0064] Perform compliance verification on each target log data and store the target log data that fails the verification in the distributed file system for use in troubleshooting.

[0065] Furthermore, the log data screening module 202 is further configured to: store the target log data that has passed the verification into a distributed database, and aggregate the target log data in the distributed database every first preset time period;

[0066] The aggregated target log data is stored in the distributed file system based on the tracking band log identification identifier.

[0067] Furthermore, the log data aggregation module 203 is also used to:

[0068] Comparing the current call link data in the relational database with the call link data in the database to be confirmed every second preset time period;

[0069] The call link data in the current call link data that is different from the call link data in the database to be confirmed is added to the database to be confirmed.

[0070] Furthermore, the call link generation module 204 is further configured to:

[0071] Based on the calling relationship between each software in the calling link data of the relational database, an interface for external use is generated.

[0072] Furthermore, the log data aggregation module 203 is specifically used to:

[0073] Determining the software included in the call link data based on the interaction log data of the request type input in each target log data;

[0074] The sequential relationship of the calls between the software is determined based on the response time of the response log data output in each target log data.

[0075] Furthermore, the call link generation module 204 is further configured to adjust the call link data based on the transaction identifier generated on the call link.

[0076] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0077] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0078] 3 , an embodiment of the present disclosure further provides an electronic device, which may include: a memory 301 and a processor 302 .

[0079] The memory 301 is used to store programs.

[0080] The processor 302 is configured to execute the program to implement the various steps of the log processing method described in the above embodiment.

[0081] An embodiment of the present disclosure further provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the log processing method described in the above embodiment are implemented.

[0082] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0083] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

Claims

1. A log processing method, characterized in that: include: Obtain the original log data of each tracking band based on the message queue; Filtering target log data containing corresponding tracking band log identification identifiers from each original log data; Aggregating the target log data based on the tracking band log identification identifier of each target log data to obtain call link data; After associating the obtained call link data with the corresponding tracking log identifier, it is stored in a preset relational database.

2. The log processing method according to claim 1, characterized in that: After filtering out the target log data containing the corresponding tracking band log identification identifier from each original log data, it also includes: Perform compliance verification on each target log data and store the target log data that fails the verification in the distributed file system for use in troubleshooting.

3. The log processing method according to claim 2, characterized in that: Also includes: Storing the verified target log data in a distributed database, and aggregating the target log data in the distributed database every first preset time period; The aggregated target log data is stored in the distributed file system based on the tracking band log identification identifier.

4. The log processing method according to claim 1, wherein: Also includes: Comparing the current call link data in the relational database with the call link data in the database to be confirmed every second preset time period; The call link data in the current call link data that is different from the call link data in the database to be confirmed is added to the database to be confirmed.

5. The log processing method according to claim 1, wherein: Also includes: Based on the calling relationship between each software in the calling link data of the relational database, an interface for external use is generated.

6. The log processing method according to claim 1, wherein: The tracking band log identification identifier of each target log data is used to aggregate the target log data to obtain call link data, including: Determining the software included in the call link data based on the interaction log data of the request type input in each target log data; The sequential relationship of the calls between the software is determined based on the response time of the response log data output in each target log data.

7. The log processing method according to claim 1, wherein: Also includes: The call link data is adjusted based on the transaction identifier generated on the call link.

8. A log processing device, characterized in that: include: The log data acquisition module is used to obtain the original log data of each tracking band based on the message queue; The log data screening module is used to screen out target log data containing corresponding tracking band log identification marks from each original log data; A log data aggregation module is used to aggregate the target log data based on the tracking band log identification identifier of each target log data to obtain call link data; as well as The call link generation module is used to associate the obtained call link data with the corresponding tracking log identifier and store them in a preset relational database.

9. An electronic device, characterized in that: include: memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the log processing method according to any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the log processing method according to any one of claims 1 to 7 is implemented.

Images