Error log analysis method and device and computer equipment

Abstract

The invention relates to an error log analysis method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: under the condition that an external request is received through a gateway and the external request is a legal request, acquiring a global identifier from a log acquisition service through the gateway; and adding the transmission packet containing the global identifier to the external request through the gateway, and distributing the external request carrying the transmission packet to the target service node, so that the target service node processes the external request to obtain a processing result, reporting log information corresponding to the processing result and the transmission packet to a log service node corresponding to the target service node; and reporting the log information and the transmission packet to a log acquisition service through the log service node, so that the log acquisition service determines error log information and a call chain sequence corresponding to the error log information based on the global identifier in the transmission packet. By adopting the method, the analysis efficiency of the error logs can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an error log analysis method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of computer technology and Internet technology, the number of users of Internet products has increased sharply, and the number of concurrent accesses is very high. A single server can no longer serve a huge number of users at the same time. Therefore, the server can adopt the method of building a service cluster to improve the system throughput and ensure the redundancy of the service. With the advantages of cloud services and the continuous improvement of container cluster deployment tools, microservices are shining, and more and more companies are using microservices and enjoying the convenience brought by microservices.

[0003] However, in the current error log analysis method, especially in the scenario of using microservices, the numerous services lead to a long and complex call chain. Once an error occurs in a module, it is necessary to manually query upwards based on the error. This process is complicated and time-consuming. If the error code is not written in a standardized manner, it may sometimes be impossible to query the root cause based on the error, which brings great trouble to R&D development and operation and maintenance work. Therefore, how to improve the analysis efficiency of error logs has become an urgent problem to be solved. Summary of the invention

[0004] Based on this, it is necessary to provide an error log analysis method, apparatus, computer device, computer readable storage medium and computer program product to address the above technical problems, which can effectively improve the error log analysis efficiency.

[0005] In a first aspect, the present application provides an error log analysis method, comprising: when an external request is received through a gateway and the external request is a legitimate request, obtaining a global identifier from a log collection service through the gateway; adding a transfer package containing the global identifier to the external request through the gateway, and distributing the external request carrying the transfer package to a target service node, so that the target service node processes the external request to obtain a processing result, and reports log information and the transfer package corresponding to the processing result to a log service node corresponding to the target service node; reporting the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package.

[0006] In one of the embodiments, the method further includes: when an external request is received through a gateway and the external request is an illegal request, determining whether a client identifier corresponding to the external request is located in a priority queue; and when the client identifier corresponding to the external request is not located in the priority queue, ignoring the external request.

[0007] In one of the embodiments, the method also includes: when the client identifier corresponding to the external request is in the priority queue, reporting the error information corresponding to the external request to the log collection service through the gateway, so that the log collection service determines whether to perform persistent storage or visual display according to the request content corresponding to the external request.

[0008] In one of the embodiments, after obtaining the global identifier from the log collection service through the gateway, the method further includes: adding the global identifier to the transmission package; processing the external request to obtain the processing result, including: when processing the external request and needing to call other service nodes, obtaining the request arrival time of the external request, the forwarding time of forwarding the external request and the node information of the external request itself; adding the request arrival time, the forwarding time and the node information to the transmission package to obtain an updated transmission package, and forwarding the external request carrying the updated transmission package to the other service nodes for processing to obtain the processing result.

[0009] In one of the embodiments, the processing result includes a successful processing result and a failed processing result; reporting the log information corresponding to the processing result and the transfer package to the log service node corresponding to the target service node includes: when the processing result is a successful processing result, reporting the log information corresponding to the successful processing result to the log service node corresponding to the target service node; when the processing result is a failed processing result, reporting the error log information corresponding to the failed processing result and the update transfer package to the log service node corresponding to the target service node; wherein the update transfer package includes the global identifier, the request arrival time, the forwarding time and the node information.

[0010] In one of the embodiments, the error log information and the call chain sequence corresponding to the error log information are determined based on the global identifier in the transmission package, including: based on the global identifier in the update transmission package, obtaining the error log information with the same global identifier; according to the request arrival time, the forwarding time and the node information in the update transmission package, determining the call chain sequence corresponding to the error log information; the method also includes: persistently storing or visually displaying the error log information and the call chain sequence.

[0011] In one of the embodiments, after determining the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package, the method also includes: releasing the global identifier in the transfer package so that the global identifier is used to be added to other external requests.

[0012] In the second aspect, the present application also provides an error log analysis device, including: an acquisition module, used to obtain a global identifier from a log collection service through a gateway when an external request is received through a gateway and the external request is a legitimate request; an adding module, used to add a transfer package containing the global identifier to the external request through the gateway; a distribution module, used to distribute the external request carrying the transfer package to a target service node, so that the target service node processes the external request to obtain a processing result, and reports the log information corresponding to the processing result and the transfer package to the log service node corresponding to the target service node; a reporting module, used to report the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package.

[0013] In a third aspect, the present application also provides a computer device, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program: when an external request is received through a gateway and the external request is a legitimate request, obtaining a global identifier from a log collection service through the gateway; adding a transfer package containing the global identifier to the external request through the gateway, and distributing the external request carrying the transfer package to a target service node, so that the target service node processes the external request to obtain a processing result, and reports the log information and the transfer package corresponding to the processing result to the log service node corresponding to the target service node; reporting the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package.

[0014] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the following steps when executed by a processor: when an external request is received through a gateway and the external request is a legitimate request, obtaining a global identifier from a log collection service through the gateway; adding a transfer package containing the global identifier to the external request through the gateway, and distributing the external request carrying the transfer package to a target service node, so that the target service node processes the external request to obtain a processing result, and reports the log information and the transfer package corresponding to the processing result to the log service node corresponding to the target service node; reporting the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package.

[0015] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps: when an external request is received through a gateway and the external request is a legitimate request, obtaining a global identifier from a log collection service through the gateway; adding a transfer package containing the global identifier to the external request through the gateway, and distributing the external request carrying the transfer package to a target service node, so that the target service node processes the external request to obtain a processing result, and reports the log information and the transfer package corresponding to the processing result to the log service node corresponding to the target service node; reporting the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package.

[0016] The above-mentioned error log analysis method, device, computer equipment, computer-readable storage medium and computer program product, when an external request is received through a gateway and the external request is a legitimate request, obtains a global identifier from a log collection service through the gateway, further adds a transfer package containing the global identifier to the external request through the gateway, and distributes the external request carrying the transfer package to a target service node, so that the target service node processes the external request to obtain a processing result, and reports log information and a transfer package corresponding to the processing result to a log service node corresponding to the target service node; reports the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package. Since the global identifier is obtained by the gateway from the log collection service in real time, the gateway can automatically add the transfer package containing the global identifier to the external request, and distribute the external request carrying the transfer package to the target service node, so that the target service node processes the external request to obtain the processing result, and reports the log information and transfer package corresponding to the processing result to the log service node corresponding to the target service node, so that the log collection service can automatically determine the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package, which can achieve accurate tracking of the calling sequence of microservices, realize automatic discovery, reporting and processing of error logs, and effectively improve the analysis efficiency of error logs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] FIG1 is a diagram showing an application environment of an error log analysis method according to an embodiment;

[0019] FIG2 is a schematic diagram of a flow chart of a method for analyzing an error log in one embodiment;

[0020] FIG3 is an architectural diagram of the overall system processing of an error log analysis method provided in one embodiment;

[0021] FIG4 is a structural block diagram of an error log analysis device according to an embodiment;

[0022] FIG. 5 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] The error log analysis method provided by the embodiment of the present application can be applied to the application environment as shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. When the server 104 receives an external request sent by the terminal 102 through the gateway, and the external request is a legal request, the server 104 obtains the global identifier from the log collection service through the gateway; the server 104 adds the transfer package containing the global identifier to the external request through the gateway, and distributes the external request carrying the transfer package to the target service node, so that the target service node processes the external request to obtain the processing result, and reports the log information and transfer package corresponding to the processing result to the log service node corresponding to the target service node; the server 104 reports the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0025] In an exemplary embodiment, as shown in FIG2 , a method for analyzing an error log is provided, which is described by taking the method applied to the server in FIG1 as an example, and includes the following steps 202 to 206. Among them:

[0026] Step 202: When an external request is received through the gateway and the external request is a legitimate request, a global identifier is obtained from a log collection service through the gateway.

[0027] Among them, external requests refer to requests sent by different clients. For example, the external requests in this application can be business requests sent by different clients (different users).

[0028] A legitimate request is a request that passes the legitimacy check, which means that the request is a legitimate request.

[0029] The log collection service refers to a service used for log collection and analysis. For example, the log collection service in the present application may be a service LogCollection specifically used for log collection and analysis, and the LogCollection may be an independently deployed server.

[0030] The global identifier is an identifier used to identify a unique request. For example, in this application, the global identifier can be a globally unique Sequence requested by the gateway from the LogCollection. The Sequence is used to uniquely mark the current request and is added to the transmission package.

[0031] Optionally / exemplarily, as shown in FIG3, it is an architectural diagram of the overall system processing of the error log analysis method provided by the present application. Take the application of this method to a microservice cluster as an example for illustration. As shown in FIG3, the microservice cluster can receive external requests sent by different clients through a gateway (Nginx), and verify the legitimacy of the external request. If the external request passes the legitimacy verification, it means that the external request is a legal request. In the case where the external request is a legal request, the microservice cluster can obtain a global identifier from the log collection service (LogCollection server) through the gateway (Nginx) to identify the unique external request. For example, as shown in FIG3, when an external request A enters, the Nginx gateway determines whether the external request A is legal. If the external request A is illegal, the Nginx gateway continues to determine whether the client A (client identifier or user identifier) ​​to which the external request A belongs is in the priority queue; if it is in the priority queue, the illegal error information of the external request A is directly reported to the log collection service, i.e., LogCollection, through the Nginx gateway. Since the error message at this time carries less information, LogCollection can decide whether to store the error message persistently or output it to WebDashboard for visual display based on the specific request content. If the external request A is legal, the Nginx gateway applies to LogCollection for a globally unique Sequence to uniquely mark the current request, that is, the external request A.

[0032] Step 204, adds the transfer package containing the global identifier to the external request through the gateway, and distributes the external request carrying the transfer package to the target service node, so that the target service node processes the external request to obtain the processing result, and reports the log information and transfer package corresponding to the processing result to the log service node corresponding to the target service node.

[0033] The transfer package refers to a tool for transferring information. For example, the transfer package in this application can be a Context package based on the Context mechanism, that is, a Context mechanism similar to the Go language is used. The transfer package containing the global identifier means: adding the global identifier to the transfer package so that the transfer package contains the global identifier.

[0034] The target service node refers to a service node in the microservice cluster, which may also be a microservice cluster. For example, the target service node in the present application may be Servers Cluster 1 as shown in FIG. 3 , where Servers Cluster 1 is one of the service (cluster) nodes in the microservice cluster.

[0035] A log service node refers to a service node used to collect log information sent by each microservice in a service cluster. For example, the log service node in the present application may be a LogServer in each service node cluster as shown in FIG3 . The LogServer is responsible for collecting error log information sent by each microservice in the cluster, that is, it is necessary to identify and filter out the error information corresponding to each different request, and integrate and further report the error information to the LogServer (such as Middle LogServer) or LogCollection of the upper call chain.

[0036] Specifically, after the microservice cluster obtains the global identifier from the log collection service through the gateway, the microservice cluster can add the transfer package containing the global identifier to the external request through the gateway, and distribute the external request carrying the transfer package to the target service node, so that the target service node processes the external request to obtain the processing result, and reports the log information and transfer package corresponding to the processing result to the log service node corresponding to the target service node. For example, as shown in Figure 3, the microservice cluster can add the transfer package Context containing the global identifier Sequence to the external request A through the gateway Nginx, and distribute the external request A carrying the transfer package Context to the target service node Servers Cluster1, so that the target service node Servers Cluster1 processes the external request A to obtain the processing result A, and reports the log information A and the transfer package Context corresponding to the processing result A to the log service node LogServer corresponding to the target service node ServersCluster1. Among them, the log service node LogServer has an association relationship with the target service node Servers Cluster1, that is, each microservice in the target service node Servers Cluster1 reports the log information and transmission package Context corresponding to the processing result to the log service node LogServer, and the log service node LogServer can be one of the service nodes belonging to the target service node Servers Cluster1.

[0037] Step 206: Report the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package.

[0038] Among them, error log information refers to the error log information generated when the microservice cluster processes external requests.

[0039] The call chain sequence corresponding to the error log information means that when the microservice cluster processes external requests, different service nodes need to be called in a certain order internally to complete the processing of the request. The call chain sequence corresponding to the error log information is the call chain sequence generated by calling different service nodes internally.

[0040] Specifically, the microservice cluster adds a transfer package containing a global identifier to an external request through a gateway, and distributes the external request carrying the transfer package to a target service node, so that the target service node processes the external request to obtain a processing result, and reports the log information and transfer package corresponding to the processing result to the log service node corresponding to the target service node. After that, the microservice cluster can further report the log information and transfer package to the log service node of the upper-level call chain through the log service node. If there is no log service node of the upper-level call chain, the log information and transfer package are directly reported to the log collection service, so that the log collection service searches for error log information with the same global identifier from all reported log information based on the global identifier in the transfer package, and determines the call chain order corresponding to the error log information according to the request arrival time, forwarding time and node information in the transfer package.

[0041] For example, as shown in Figure 3, the microservice cluster adds the transfer package Context containing the global identifier Sequence1 to the external request A through the gateway Nginx, and distributes the external request A carrying the transfer package Context to the target service node Servers Cluster1, so that the target service node Servers Cluster1 processes the external request A to obtain the processing result A, and reports the log information A and the transfer package Context corresponding to the processing result A to the log service node LogServer1 corresponding to the target service node Servers Cluster1. After that, the microservice cluster reports the collected log information and transfer package Context to the LogServer or log collection service of the upper call chain, i.e., LogCollection, through the log service node LogServer1, so that the log collection service LogCollection searches for the error log information with the same global identifier Sequence1 from all the reported log information based on the global identifier Sequence1 in the transfer package Context, and determines the call chain sequence corresponding to the error log information according to the request arrival time, forwarding time and node information in the transfer package Context. For example, assuming that the calling sequence between the internal service nodes of the external request A is node A→node B→node C, the log collection service LogCollection searches for error log information with the same global identifier Sequence1 from all reported log information based on the global identifier Sequence1 in the transmission package Context, and determines the call chain sequence corresponding to the error log information as A→node B→node C according to the request arrival time, forwarding time and node information in the transmission package Context, and outputs it to the WebDashboard for visual display or persistence in the database, that is, visualizes the call link between microservices to achieve accurate tracking, facilitates process tracking and problem troubleshooting, and makes error location more accurate and intuitive.

[0042] In this embodiment, when an external request is received through the gateway and the external request is a legitimate request, the global identifier is obtained from the log collection service through the gateway. Furthermore, the transfer package containing the global identifier is added to the external request through the gateway, and the external request carrying the transfer package is distributed to the target service node, so that the target service node processes the external request to obtain a processing result, and reports the log information and transfer package corresponding to the processing result to the log service node corresponding to the target service node; the log information and transfer package are reported to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package. Since the global identifier is obtained by the gateway from the log collection service in real time, the gateway can automatically add the transfer package containing the global identifier to the external request, and distribute the external request carrying the transfer package to the target service node, so that the target service node processes the external request to obtain the processing result, and reports the log information and transfer package corresponding to the processing result to the log service node corresponding to the target service node, so that the log collection service can automatically determine the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package, which can achieve accurate tracking of the calling sequence of microservices, realize automatic discovery, reporting and processing of error logs, and effectively improve the analysis efficiency of error logs.

[0043] In an exemplary embodiment, the method further comprises:

[0044] When an external request is received through the gateway and the external request is an illegal request, determining whether the client identifier corresponding to the external request is in the priority queue;

[0045] If the client identifier corresponding to the external request is not in the priority queue, the external request is ignored.

[0046] Specifically, the method is applied to a microservice cluster as an example for illustration. As shown in FIG3 , the microservice cluster can receive external requests sent by different clients through the gateway (Nginx) and verify the legitimacy of the external requests. For example, when an external request A enters, the Nginx gateway is first used to determine whether the external request A is legal. If the external request A is not legal, the Nginx gateway is further used to determine whether the client A (client identifier or user identifier) ​​to which the external request A belongs is in the priority queue; if it is in the priority queue, the Nginx gateway directly reports the illegal error information of the external request A to the log collection service, i.e., LogCollection. Since the amount of information carried by the error information at this time is small, LogCollection can be used to decide whether to persist the error information or output it to the Dashboard for visual display according to the specific request content; if the client identifier A corresponding to the external request A is not in the priority queue, the gateway can directly ignore the external request A, i.e., do not process it. As a result, the error point can be quickly located according to the call chain and error log information, achieving efficient development and operation and maintenance work, which will greatly reduce manpower and time costs.

[0047] In an exemplary embodiment, the method further comprises:

[0048] When the client identifier corresponding to the external request is in the priority queue, the error information corresponding to the external request is reported to the log collection service through the gateway, so that the log collection service determines whether to perform persistent storage or visual display based on the request content corresponding to the external request.

[0049] Specifically, the method is applied to a microservice cluster as an example for illustration. As shown in FIG3 , the microservice cluster can receive external requests sent by different clients through a gateway (Nginx) and verify the legitimacy of the external requests. For example, when an external request A enters, the Nginx gateway is first used to determine whether the external request A is legal. If the external request A is illegal, the Nginx gateway is used to determine whether the client A (client ID or user ID) to which the external request A belongs is in the priority queue; if it is in the priority queue, the Nginx gateway directly reports the illegal error information of the external request A to the log collection service, i.e., LogCollection. Since the amount of information carried by the error information at this time is small, LogCollection can be used to decide whether to persist the error information or output it to the Dashboard for visual display according to the specific request content. As a result, through the analysis of the aggregated error logs, LogServer can output the most accurate error logs, and can be set to achieve large-screen display and database persistence, which greatly improves the error troubleshooting capability, improves the development and operation efficiency, reduces the manpower and time cost, and can effectively improve the analysis efficiency of the error logs.

[0050] In an exemplary embodiment, after obtaining the global identifier from the log collection service through the gateway, the method further includes:

[0051] Add global identification to the transmittal package;

[0052] The processing of the external request to obtain a processing result includes:

[0053] When processing external requests and needing to call other service nodes, obtain the request arrival time of the external request, the forwarding time of the external request, and the node information of the node itself;

[0054] The request arrival time, forwarding time and node information are added to the transfer package to obtain an updated transfer package, and the external request carrying the updated transfer package is forwarded to other service nodes for processing to obtain a processing result.

[0055] Specifically, as shown in Figure 3, the microservice cluster can add the transmission package Context containing the global identifier Sequence1 to the external request A through the gateway Nginx, and distribute the external request A carrying the transmission package Context to the target service node Servers Cluster1, so that the target service node Servers Cluster1 processes the external request A to obtain the processing result A, and reports the log information A corresponding to the processing result A and the transmission package Context to the log service node LogServer corresponding to the target service node Servers Cluster1. Among them, when node A in the target service node Servers Cluster1 processes the external request A and needs to call other service nodes (for example, node B in the service node ServersCluster2 needs to be called), node A in the target service node Servers Cluster1 can obtain the request arrival time of the external request A, the forwarding time of the external request, and its own node information (i.e., node A), and add the request arrival time, forwarding time, and node information (i.e., node A) to the transmission package Context to obtain the updated transmission package Context1, and forward the external request A carrying the updated transmission package Context1 to other service nodes, i.e., node B in ServersCluster1 for processing to obtain the processing result B, and return the processing result to node A in the target service node Servers Cluster1. That is, the calling order between the internal service nodes of the external request A is node A→node B. Therefore, by adopting the call chain tracking mechanism based on Context, it is possible to accurately track the calling order of microservices, realize the automatic discovery, reporting, and processing of error logs, and effectively improve the analysis efficiency of error logs.

[0056] In an exemplary embodiment, the processing result includes a processing success result and a processing failure result; and the step of reporting log information and a transfer package corresponding to the processing result to a log service node corresponding to the target service node includes:

[0057] When the processing result is a successful result, the log information corresponding to the successful processing result is reported to the log service node corresponding to the target service node;

[0058] When the processing result is a processing failure, the error log information and update transmission package corresponding to the processing failure result are reported to the log service node corresponding to the target service node; wherein the update transmission package includes the global identifier, request arrival time, forwarding time and node information.

[0059] Specifically, as shown in Figure 3, the microservice cluster can add the transmission package Context containing the global identifier Sequence1 to the external request A through the gateway Nginx, and distribute the external request A carrying the transmission package Context to the target service node Servers Cluster1, so that the target service node Servers Cluster1 processes the external request A to obtain the processing result A, and reports the log information A corresponding to the processing result A and the transmission package Context to the log service node LogServer corresponding to the target service node Servers Cluster1. That is, when the processing result A is a successful processing result, the target service node Servers Cluster1 reports the log information A and the transmission package Context (or the updated transmission package Context) corresponding to the successful processing result to the log service node LogServer corresponding to the target service node Servers Cluster1, that is, reports the success information corresponding to the external request A. At this time, the transmission package Context only needs to include the Sequence1 information; when the processing result A is a failed processing result, the target service node Servers Cluster1 reports the error log information and the updated transmission package Context (or the transmission package Context) corresponding to the failed processing result to the log service node LogServer corresponding to the target service node Servers Cluster1; wherein the updated transmission package Context includes the global identifier, the request arrival time, the forwarding time and the node information, so that the subsequent log collection service, i.e., LogCollection, can search for the error log information with the same global identifier Sequence1 from all the reported log information based on the global identifier Sequence1 in the updated transmission package, and determine the call chain sequence corresponding to the error log information according to the request arrival time, the forwarding time and the node information in the updated transmission package Context. Therefore, by adopting the Context-based call chain tracking mechanism, it is possible to accurately track the calling sequence of microservices, realize automatic discovery, reporting and processing of error logs, and effectively improve the analysis efficiency of error logs.

[0060] In an exemplary embodiment, the step of determining the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transmission package includes:

[0061] Based on the global identifier in the update delivery package, obtaining error log information having the same global identifier;

[0062] Determine the call chain sequence corresponding to the error log information according to the request arrival time, forwarding time and node information in the update transfer packet;

[0063] The method also includes: persistently storing or visually displaying the error log information and the call chain sequence.

[0064] Specifically, the log collection service LogCollection shown in FIG3 can search for error log information with the same global identifier Sequence1 from all reported log information based on the global identifier Sequence1 in the update transfer package Context, and determine the call chain sequence corresponding to the error log information according to the request arrival time, forwarding time and node information in the update transfer package Context. For example, assuming that the call sequence between the internal service nodes of the external request A is node A→node B→node C, the log collection service LogCollection searches for error log information with the same global identifier Sequence1 from all reported log information based on the global identifier Sequence1 in the update transfer package Context, and determines the call chain sequence corresponding to the error log information as A→node B→node C according to the request arrival time, forwarding time and node information in the update transfer package Context, and outputs it to WebDashboard for visual display or database persistence, that is, visualizes the call link between microservices, realizes accurate tracking, facilitates process tracking and problem troubleshooting, and makes error location more accurate and intuitive.

[0065] In an exemplary embodiment, after determining the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer packet, the method further includes:

[0066] Releases the global ID in the transmittal package so that the global ID can be used to add to other external requests.

[0067] Specifically, as shown in FIG3 , the log collection service LogCollection searches for error log information with the same global identifier Sequence1 from all reported log information based on the global identifier Sequence1 in the update delivery package Context, and determines the call chain sequence corresponding to the error log information according to the request arrival time, forwarding time and node information in the update delivery package Context. For example, assuming that the call sequence between the internal service nodes of the external request A is node A→node B→node C, the log collection service LogCollection searches for error log information with the same global identifier Sequence1 from all reported log information based on the global identifier Sequence1 in the update delivery package Context, and determines the call chain sequence corresponding to the error log information as A→node B→node C according to the request arrival time, forwarding time and node information in the update delivery package Context, and outputs it to the WebDashboard for visual display or database persistence, and then the log collection service LogCollection automatically releases the global identifier Sequence1 in the update delivery package Context, so that the global identifier Sequence1 is used to add to other external requests, that is, the log collection service LogCollection releases the Sequence and uses it again for other requests. As a result, through the visual call link and automatic error log analysis mechanism, front-line operation and maintenance can first check, and then R&D can intervene to check, which can effectively reduce the workload of R&D and improve operation and maintenance efficiency.

[0068] In an exemplary embodiment, the method provided in the embodiment of the present application can be applied to the analysis of error logs in various scenarios. The following describes the error log analysis method provided in the embodiment of the present application by taking the application in a microservice cluster as an example.

[0069] Problems and disadvantages of traditional methods include:

[0070] Today's Internet is undergoing a technological revolution. As cloud native becomes more mature and related tools continue to emerge, more and more companies are embracing the cloud native era, and everything on the cloud is slowly becoming a reality. Since the launch of Docker, with the advantages of cloud services and the continuous improvement of container cluster deployment tools, it has shined in microservices and has been accepted by most companies. However, while enjoying the convenience brought by microservices, problems also arise. The numerous services lead to long and complex call chains. Once an error occurs in a module, it is necessary to continuously query upwards based on the error. This process is complicated and time-consuming. If the error code is not written in a standardized manner, sometimes it may not be possible to query the root cause based on the error, which brings great trouble to the development and operation of R&D.

[0071] Therefore, in order to solve the above problems, an automatic error log analysis method based on Context is proposed in the embodiment of the present application, which is similar to the Context mechanism of the Go language. The Context is continuously used in the call chain to track and trace the call results, and the error log is automatically discovered, reported and processed. Finally, the error reporting information on the entire call chain is summarized and integrated to perform WebDashboard display or database persistence operations.

[0072] Among them, Nginx gateway: Nginx, as a gateway, is responsible for filtering and forwarding access requests. It can be combined with other gateway tools as needed to achieve richer and more complex gateway functions, such as multiple gateway integration, etc. For the sake of brevity, it will not be repeated here.

[0073] LogCollection is a log collection and analysis service. It is responsible for receiving error log information sent by all LogServers, integrating and summarizing the information, and displaying the final results through WebDashboard or performing persistence operations.

[0074] Servers Clusters: represents the microservice clusters on each node, which are responsible for actual business processing. It continuously calls other microservices according to the actual request until the entire request is processed. When an error occurs, the error information will be reported to the LogServer service in the cluster, which will complete the log analysis, integration and further reporting.

[0075] LogServer: Responsible for collecting error log information sent by each microservice in the cluster. It is necessary to identify and filter out the error information corresponding to each different request, integrate these error information, and further report them to the LogServer or LogCollection of the upper call chain.

[0076] The purpose of this application to propose a context-based error log automatic analysis method is to:

[0077] 1. Accurately track the call chain between microservices: visualize the call chain between microservices and accurately track it to facilitate process tracking and problem troubleshooting.

[0078] 2. Reduce the cost of error troubleshooting: Through the visual call link and automatic error log analysis mechanism, front-line operation and maintenance can first troubleshoot, and then R&D can intervene to troubleshoot. This can effectively reduce the workload of R&D and improve operation and maintenance efficiency.

[0079] 3. Improve real-time monitoring of each microservice: Through the visualization of the large screen, you can accurately know the specific call chain of a microservice that has a problem, rather than just staying at the macro data statistics. The granularity of error display is further reduced, and error positioning is more accurate and intuitive.

[0080] As shown in FIG3 , the present application proposes a context-based error log automatic analysis method, and the specific implementation process is as follows:

[0081] 1. When an external request comes in, the Nginx gateway makes a judgment. If the request is illegal, it continues to determine whether the user to which the connection belongs is in the priority queue. If it is in the priority queue, the error information is reported to LogCollection. The amount of information carried by the error information at this time is small. LogCollection decides whether to persist or output it to the Dashboard based on the specific request content; otherwise, the illegal request is ignored and no error is reported to LogCollection. If the request is legal, Nginx will apply to LogCollection for a globally unique Sequence to uniquely mark the current request and add it to the Context. The Context information will be added to the current request. The Context will also contain information such as the request arrival time and forwarding time, the last processed node information, etc. After the Context information is added, proceed to step 2.

[0082] 2. At this time, the request will be processed by a service in the microservice cluster. If the current service can process the request content without error, it will return the request result, end all processes of the current request, and report success information to the LogServer corresponding to the current cluster. At this time, the Context only needs to contain Sequence information, and the LogServer will continue to report until the LogCollection, which will release the Sequence and use it again for other requests. If the processing fails, the error information and the corresponding Context will be reported to the LogServer corresponding to the current cluster. The LogServer will integrate the error information and send it to the upper-level LogServer or LogCollection (if there is no upper-level LogServer). When the LogCollection is processed, the current Sequence will be released. If the current service needs to further call the services of other clusters to complete the processing of the request, update the Context and append the time and service node information when the current node forwards the request. Repeat step 2.

[0083] 3. LogCollection will first divide all received error information according to Sequence; then, LogCollection will summarize and sort out the complete error log information with the correct call chain sequence for each identical information of each Sequence according to the call and forwarding time and the service node information, and output it to the WebDashboard or perform persistence operations in the database.

[0084] Through the above steps, developers and operation and maintenance personnel will be able to easily troubleshoot numerous microservice errors. They can quickly locate the error point based on the call chain and error log information, achieve high-speed and efficient development and operation and maintenance work, and greatly reduce manpower and time costs.

[0085] The beneficial effects of the technical solution of this application include:

[0086] 1. A Context-based call chain tracking mechanism is proposed to accurately track the microservice call sequence.

[0087] 2. By analyzing the aggregated error logs, LogServer can output the most accurate error logs, and can be set up to display them on a large screen and persist them in a database.

[0088] 3. Development and operation personnel can perform accurate error troubleshooting based on accurate error logs, greatly improving error troubleshooting capabilities, improving development and operation efficiency, and reducing manpower and time costs.

[0089] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0090] Based on the same inventive concept, the embodiment of the present application also provides an error log analysis device for implementing the error log analysis method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more error log analysis devices provided below can refer to the limitations of the error log analysis method above, and will not be repeated here.

[0091] In an exemplary embodiment, as shown in FIG. 4 , an error log analysis device is provided, including: an acquisition module 402 , an adding module 404 , a distribution module 406 and a reporting module 408 , wherein:

[0092] The acquisition module 402 is used to acquire a global identifier from a log collection service through the gateway when an external request is received through the gateway and the external request is a legal request.

[0093] The adding module 404 is used to add the transfer package including the global identifier to the external request through the gateway.

[0094] The distribution module 406 is used to distribute the external request carrying the transfer package to the target service node, so that the target service node processes the external request to obtain a processing result, and reports the log information corresponding to the processing result and the transfer package to the log service node corresponding to the target service node.

[0095] The reporting module 408 is used to report the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package.

[0096] In one embodiment, the device also includes: a judgment module, which is used to determine whether the client identifier corresponding to the external request is located in the priority queue when the external request is received through the gateway and the external request is an illegal request; and an ignoring module, which is used to ignore the external request when the client identifier corresponding to the external request is not located in the priority queue.

[0097] In one embodiment, the reporting module is also used to report the error information corresponding to the external request to the log collection service through the gateway when the client identifier corresponding to the external request is in the priority queue, so that the log collection service determines whether to perform persistent storage or visual display based on the request content corresponding to the external request.

[0098] In one embodiment, the adding module is also used to add the global identifier to the transmission package; the obtaining module is also used to obtain the request arrival time of the external request, the forwarding time of the external request and its own node information when processing the external request and needing to call other service nodes; the adding module is also used to add the request arrival time, the forwarding time and the node information to the transmission package to obtain an updated transmission package, and forward the external request carrying the updated transmission package to the other service nodes for processing to obtain the processing result.

[0099] In one embodiment, the processing result includes a successful processing result and a failed processing result; the reporting module is also used to report the log information corresponding to the successful processing result to the log service node corresponding to the target service node when the processing result is a successful processing result; when the processing result is a failed processing result, report the error log information corresponding to the failed processing result and the update transmission package to the log service node corresponding to the target service node; wherein the update transmission package includes the global identifier, the request arrival time, the forwarding time and the node information.

[0100] In one embodiment, the acquisition module is also used to obtain error log information with the same global identifier based on the global identifier in the update transmission package; the device also includes: a determination module, used to determine the call chain sequence corresponding to the error log information according to the request arrival time, the forwarding time and the node information in the update transmission package; a processing module, used to persistently store or visually display the error log information and the call chain sequence.

[0101] Each module in the above error log analysis device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0102] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store analysis data of the error log. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an error log analysis method is implemented.

[0103] Those skilled in the art will understand that the structure shown in FIG. 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0104] In an exemplary embodiment, in one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0105] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0106] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0107] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0108] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0109] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0110] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for analyzing an error log, characterized in that: The method comprises: When an external request is received through the gateway and the external request is a legitimate request, obtaining a global identifier from a log collection service through the gateway; Adding a transfer package containing the global identifier to the external request through the gateway, and distributing the external request carrying the transfer package to a target service node, so that the target service node processes the external request to obtain a processing result, and reports log information corresponding to the processing result and the transfer package to a log service node corresponding to the target service node; The log information and the transfer package are reported to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package.

2. The method according to claim 1, characterized in that: The method further comprises: When an external request is received through the gateway and the external request is an illegal request, determining whether the client identifier corresponding to the external request is in a priority queue; If the client identifier corresponding to the external request is not in the priority queue, the external request is ignored.

3. The method according to claim 2, characterized in that The method further comprises: When the client identifier corresponding to the external request is in the priority queue, the error information corresponding to the external request is reported to the log collection service through the gateway, so that the log collection service determines whether to perform persistent storage or visual display according to the request content corresponding to the external request.

4. The method according to claim 1, characterized in that: After obtaining the global identifier from the log collection service through the gateway, the method further includes: adding the global identifier to the delivery package; The processing of the external request to obtain a processing result includes: When processing the external request and needing to call other service nodes, obtaining the request arrival time of the external request, the forwarding time of the external request and the node information of the node itself; The request arrival time, the forwarding time and the node information are added to the delivery package to obtain an updated delivery package, and the external request carrying the updated delivery package is forwarded to the other service node for processing to obtain the processing result.

5. The method according to claim 4, characterized in that The processing result includes a successful processing result and a failed processing result; The reporting of the log information corresponding to the processing result and the transmission package to the log service node corresponding to the target service node includes: When the processing result is a successful processing result, reporting log information corresponding to the successful processing result to the log service node corresponding to the target service node; When the processing result is a processing failure, the error log information corresponding to the processing failure result and the update transmission package are reported to the log service node corresponding to the target service node; wherein the update transmission package includes the global identifier, the request arrival time, the forwarding time and the node information.

6. The method according to claim 5, characterized in that The step of determining the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transmission package includes: Based on the global identifier in the update delivery package, acquiring error log information having the same global identifier; Determine the call chain sequence corresponding to the error log information according to the request arrival time, the forwarding time, and the node information in the update delivery packet; The method further includes: persistently storing or visually displaying the error log information and the call chain sequence.

7. An error log analysis device, characterized in that: The device comprises: An acquisition module, configured to acquire a global identifier from a log collection service through the gateway when an external request is received through the gateway and the external request is a legitimate request; An adding module, used for adding a delivery package including the global identifier to the external request through the gateway; A distribution module, used for distributing the external request carrying the transfer packet to a target service node, so that the target service node processes the external request to obtain a processing result, and reports log information corresponding to the processing result and the transfer packet to a log service node corresponding to the target service node; The reporting module is used to report the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain sequence corresponding to the error log information based on the global identifier in the transfer package.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images