Error log analysis method and device, computer device

Abstract

The application relates to an error log analysis method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: in the case that an external request is received through a gateway and the external request is a legal request, acquiring a global identifier from a log collection service through the gateway; adding a transfer package containing the global identifier to the external request through the gateway, and distributing the external request carrying the transfer package to a target service node, so that the target service node processes the external request to obtain a processing result, and reports log information corresponding to the processing result and the transfer package to a log service node corresponding to the target service node; and reporting the log information and the transfer package to the log collection service through the log service node, so that the log collection service determines error log information and a calling chain order corresponding to the error log information based on the global identifier in the transfer package. The method can effectively improve the analysis efficiency of error logs.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for analyzing error logs. Background Technology

[0002] With the development of computer and internet technologies, the number of internet product users has surged, and the number of concurrent accesses is very high. A single server can no longer serve a huge number of users simultaneously. Therefore, the server can adopt the method of building service clusters to improve system throughput and ensure service redundancy. With the advantages of cloud services and the continuous improvement of container cluster deployment tools, microservices are shining brightly, and more and more enterprises are using microservices and enjoying the convenience they bring.

[0003] However, current error log analysis methods, especially in microservice scenarios, result in long and complex call chains due to the numerous services. Once an error occurs in a module, manual querying of the error back up is required, which is tedious and time-consuming. If the error code is not written in a standardized manner, it may be impossible to find the root cause based on the error, causing great trouble for the development and operation and maintenance work. Therefore, how to improve the efficiency of error log analysis has become an urgent problem to be solved. Summary of the Invention

[0004] Therefore, it is necessary to provide an error log analysis method, apparatus, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems, which can effectively improve the analysis efficiency of error logs.

[0005] In a first aspect, this application provides a method for analyzing error logs, comprising: upon receiving an external request through a gateway, and provided that the external request is a legitimate request, obtaining a global identifier from a log collection service through the gateway; adding a transmission packet containing the global identifier to the external request through the gateway, and distributing the external request carrying the transmission packet to a target service node, so that the target service node processes the external request and obtains a processing result, and reports the log information corresponding to the processing result and the transmission packet to the log service node corresponding to the target service node; and having the log service node report the log information and the transmission packet to the log collection service, so that the log collection service determines the error log information and the call chain order corresponding to the error log information based on the global identifier in the transmission packet.

[0006] In one embodiment, the method further includes: if an external request is received through the gateway and the external request is an invalid request, determining whether the client identifier corresponding to the external request is in a priority queue; if the client identifier corresponding to the external request is not in the priority queue, ignoring the external request.

[0007] In one embodiment, the method further includes: if the client identifier corresponding to the external request is in a priority queue, reporting the error information corresponding to the external request to the log collection service through the gateway, so that the log collection service determines whether to perform persistent storage or visualization display based on the request content corresponding to the external request.

[0008] In one embodiment, after obtaining the global identifier from the log collection service through the gateway, the method further includes: adding the global identifier to the transmission packet; processing the external request to obtain a processing result includes: when processing the external request and needing to call other service nodes, obtaining the request arrival time of the external request, the forwarding time of forwarding the external request, and its own node information; adding the request arrival time, the forwarding time, and the node information to the transmission packet to obtain an updated transmission packet, and forwarding the external request carrying the updated transmission packet to the other service nodes for processing to obtain the processing result.

[0009] In one embodiment, the processing result includes a successful processing result and a failed processing result; reporting the log information corresponding to the processing result and the transmission packet to the log service node corresponding to the target service node includes: when the processing result is a successful processing result, reporting the log information corresponding to the successful processing result to the log service node corresponding to the target service node; when the processing result is a failed processing result, reporting the error log information corresponding to the failed processing result and the update transmission packet to the log service node corresponding to the target service node; wherein, the update transmission packet includes the global identifier, the request arrival time, the forwarding time, and the node information.

[0010] In one embodiment, determining the error log information and the call chain order corresponding to the error log information based on the global identifier in the transmission packet includes: obtaining error log information with the same global identifier based on the global identifier in the update transmission packet; determining the call chain order corresponding to the error log information according to the request arrival time, the forwarding time, and the node information in the update transmission packet; the method further includes: persistently storing or visually displaying the error log information and the call chain order.

[0011] In one embodiment, after determining the error log information and the call chain order corresponding to the error log information based on the global identifier in the delivery packet, the method further includes: releasing the global identifier in the delivery packet so that the global identifier can be used to add to other external requests.

[0012] Secondly, this application also provides an error log analysis apparatus, comprising: an acquisition module, configured to acquire a global identifier from a log collection service through a gateway when an external request is received through a gateway and the external request is a legitimate request; an addition module, configured to add a transmission packet containing the global identifier to the external request through the gateway; a distribution module, configured to distribute the external request carrying the transmission packet to a target service node, so that the target service node processes the external request to obtain a processing result, and reports the log information corresponding to the processing result and the transmission packet to the log service node corresponding to the target service node; and a reporting module, configured to report the log information and the transmission packet to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain order corresponding to the error log information based on the global identifier in the transmission packet.

[0013] Thirdly, this application also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: when an external request is received through a gateway and the external request is a legitimate request, a global identifier is obtained from a log collection service through the gateway; a delivery packet containing the global identifier is added to the external request through the gateway, and the external request carrying the delivery packet is distributed to a target service node, so that the target service node processes the external request and obtains a processing result, and reports the log information corresponding to the processing result and the delivery packet to the log service node corresponding to the target service node; the log service node reports the log information and the delivery packet to the log collection service, so that the log collection service determines the error log information and the call chain order corresponding to the error log information based on the global identifier in the delivery packet.

[0014] Fourthly, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the following steps: upon receiving an external request through a gateway, and provided that the external request is a legitimate request, obtaining a global identifier from a log collection service through the gateway; adding a transmission packet containing the global identifier to the external request through the gateway, and distributing the external request carrying the transmission packet to a target service node, so that the target service node processes the external request and obtains a processing result, and reports the log information corresponding to the processing result and the transmission packet to the log service node corresponding to the target service node; and reporting the log information and the transmission packet to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain order corresponding to the error log information based on the global identifier in the transmission packet.

[0015] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps: upon receiving an external request through a gateway, and provided that the external request is a legitimate request, obtaining a global identifier from a log collection service through the gateway; adding a delivery packet containing the global identifier to the external request through the gateway, and distributing the external request carrying the delivery packet to a target service node, so that the target service node processes the external request and obtains a processing result, and reports the log information corresponding to the processing result and the delivery packet to the log service node corresponding to the target service node; and reporting the log information and the delivery packet to the log collection service through the log service node, so that the log collection service determines error log information and the call chain order corresponding to the error log information based on the global identifier in the delivery packet.

[0016] The aforementioned error log analysis method, apparatus, computer equipment, computer-readable storage medium, and computer program product, upon receiving an external request through a gateway and confirming that the external request is legitimate, obtain a global identifier from the log collection service through the gateway. Further, the gateway adds a transmission packet containing the global identifier to the external request and distributes the external request carrying the transmission packet to the target service node. This allows the target service node to process the external request, obtain a processing result, and report the log information and transmission packet corresponding to the processing result to the log service node corresponding to the target service node. The log service node then reports the log information and transmission packet to the log collection service, enabling the log collection service to determine the error log information and the corresponding call chain order based on the global identifier in the transmission packet. Since the global identifier is obtained by the gateway from the log collection service in real time, the gateway can automatically add the packet containing the global identifier to the external request and distribute the external request carrying the packet to the target service node. The target service node then processes the external request, obtains the processing result, and reports the log information and packet corresponding to the processing result to the log service node corresponding to the target service node. This allows the log collection service to automatically determine the error log information and the corresponding call chain order based on the global identifier in the packet. This enables precise tracking of the microservice call order, automatic discovery, reporting, and processing of error logs, and effectively improves the efficiency of error log analysis. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a diagram illustrating the application environment of an error log analysis method in one embodiment.

[0019] Figure 2 This is a flowchart illustrating an error log analysis method in one embodiment;

[0020] Figure 3 This is an architecture diagram of the overall system processing of the error log analysis method provided in one embodiment;

[0021] Figure 4 This is a structural block diagram of an error log analysis device in one embodiment;

[0022] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] The error log analysis method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. When server 104 receives an external request from terminal 102 through a gateway, and the external request is valid, server 104 obtains a global identifier from the log collection service through the gateway. Server 104 adds a packet containing the global identifier to the external request through the gateway and distributes the external request carrying the packet to the target service node, so that the target service node processes the external request, obtains the processing result, and reports the log information and packet corresponding to the processing result to the log service node corresponding to the target service node. Server 104 reports the log information and packet to the log collection service through the log service node, so that the log collection service determines the error log information and the call chain order corresponding to the error log information based on the global identifier in the packet. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0025] In one exemplary embodiment, such as Figure 2 As shown, an error log analysis method is provided, which can be applied to... Figure 1 Taking the server in the example, the explanation includes the following steps 202 to 206. Wherein:

[0026] Step 202: If an external request is received through the gateway and the external request is a valid request, obtain the global identifier from the log collection service through the gateway.

[0027] External requests refer to requests sent by different clients. For example, in this application, external requests can be business requests sent by different clients (different users).

[0028] A legitimate request is a request that passes the legitimacy check, meaning that the request is legitimate.

[0029] Log collection service refers to a service used for log collection and analysis. For example, the log collection service in this application can be a LogCollection service specifically used for log collection and analysis, and the LogCollection can be a stand-alone server.

[0030] A global identifier is an identifier used to uniquely identify a request. For example, in this application, the global identifier can be a globally unique Sequence that the gateway requests from LogCollection. This Sequence is used to uniquely identify the current request and is added to the delivery packet.

[0031] Optionally / exemplarily, such as Figure 3 The diagram shown illustrates the overall system architecture of the error log analysis method provided in this application. The method is illustrated using an example of its application to a microservice cluster. Figure 3 As shown, a microservice cluster can receive external requests from different clients through a gateway (Nginx) and verify their validity. If the external request passes the validity verification, it is considered a legitimate request. In this case, the microservice cluster can obtain a global identifier from the log collection service (LogCollection server) through the gateway (Nginx) to uniquely identify the external request. For example, as... Figure 3 As shown, when an external request A arrives, the Nginx gateway determines whether the request is valid. If the request is invalid, the Nginx gateway checks if the client A (client identifier or user identifier) ​​to which request A belongs is in a priority queue. If it is, the Nginx gateway directly reports the invalid request error to the log collection service, LogCollection. Since the error message carries relatively little information at this stage, LogCollection can decide whether to persist the error message or display it on the WebDashboard for visualization, based on the specific request content. If the request is valid, the Nginx gateway requests a globally unique Sequence from LogCollection to uniquely identify the current request, i.e., external request A.

[0032] Step 204: Add the packet containing the global identifier to the external request through the gateway, and distribute the external request carrying the packet to the target service node so that the target service node can process the external request, obtain the processing result, and report the log information and packet corresponding to the processing result to the log service node corresponding to the target service node.

[0033] Here, a delivery package refers to a tool used to deliver information. For example, the delivery package in this application can be a Context package based on the Context mechanism, that is, it adopts a Context mechanism similar to that of the Go language. A delivery package containing a global identifier means that a global identifier is added to the delivery package so that the delivery package contains the global identifier.

[0034] The target service node refers to a specific service node within a microservice cluster. This service node can also be part of the microservice cluster itself. For example, the target service node in this application could be, for instance, a microservice cluster. Figure 3 The Servers Cluster1 shown is one of the service (cluster) nodes in the microservice cluster.

[0035] A log service node refers to a service node used to collect log information sent by various microservices within a service cluster. For example, the log service node in this application could be such as... Figure 3 The LogServer in each service node cluster shown is responsible for collecting error log information sent by various microservices in the cluster. That is, it needs to identify and filter the error information corresponding to each different request, and integrate these error information and further report them to the LogServer (such as Middle LogServer) or LogCollection in the upper-level call chain.

[0036] Specifically, after the microservice cluster obtains the global identifier from the log collection service through the gateway, it can add a packet containing the global identifier to the external request through the gateway. The external request carrying the packet is then distributed to the target service node, allowing the target service node to process the request, obtain the result, and report the corresponding log information and packet to the log service node. For example, ... Figure 3As shown, the microservice cluster can add a context packet containing a globally identifier Sequence to an external request A via the gateway Nginx. The external request A, carrying the context packet, is then distributed to the target service node Servers Cluster1. Servers Cluster1 processes the external request A, obtains the processing result A, and reports the log information A corresponding to the processing result A and the context packet to the corresponding log service node LogServer. The LogServer is associated with Servers Cluster1; each microservice in Servers Cluster1 reports the log information and context packet corresponding to the processing result to the LogServer, and the LogServer can be one of the service nodes belonging to Servers Cluster1.

[0037] Step 206: The log service node reports the log information and the transmission packet to the log collection service, so that the log collection service can determine the error log information and the call chain order corresponding to the error log information based on the global identifier in the transmission packet.

[0038] Error log information refers to the error log information generated during the microservice cluster's processing of external requests.

[0039] The call chain order corresponding to the error log information refers to the order in which different service nodes are called internally to complete the processing of the request during the process of handling external requests in a microservice cluster.

[0040] Specifically, the microservice cluster adds a packet containing a global identifier to the external request through the gateway, and distributes the external request carrying the packet to the target service node. The target service node processes the external request, obtains the processing result, and reports the log information and packet corresponding to the processing result to the log service node corresponding to the target service node. The microservice cluster can then further report the log information and packet to the log service node of the upper-level call chain through the log service node. If there is no log service node of the upper-level call chain, the log information and packet are directly reported to the log collection service. The log collection service then uses the global identifier in the packet to search for error log information with the same global identifier from all reported log information, and determines the call chain order corresponding to the error log information based on the request arrival time, forwarding time, and node information in the packet.

[0041] For example, such as Figure 3 As shown, the microservice cluster adds a Context packet containing the global identifier Sequence1 to the external request A through the gateway Nginx, and distributes the external request A carrying the Context packet to the target service node Servers Cluster1. Servers Cluster1 then processes the external request A, obtains the processing result A, and reports the log information A corresponding to the processing result A and the Context packet to the corresponding log service node LogServer1. Afterwards, the microservice cluster, through LogServer1, reports the collected log information and the Context packet to the LogServer or Log Collection service in the upper-level call chain. LogCollection then uses the global identifier Sequence1 in the Context packet to search for error log information with the same global identifier Sequence1 from all reported log information, and determines the call chain order corresponding to the error log information based on the request arrival time, forwarding time, and node information in the Context packet. For example, assuming the call order between the internal service nodes of external request A is node A → node B → node C, the log collection service LogCollection, based on the global identifier Sequence1 in the transmission packet Context, searches for error log information with the same global identifier Sequence1 from all reported log information. According to the request arrival time, forwarding time, and node information in the transmission packet Context, it determines the call chain order corresponding to the error log information as A → node B → node C, and outputs it to the WebDashboard for visualization or persists it to the database. That is, it visualizes the call chain between microservices, realizes accurate tracking, facilitates process tracking and problem investigation, and makes error location more accurate and intuitive.

[0042] In this embodiment, when an external request is received through the gateway and the external request is valid, the gateway obtains a global identifier from the log collection service. Further, the gateway adds a transmission packet containing the global identifier to the external request and distributes the external request carrying the transmission packet to the target service node. This allows the target service node to process the external request, obtain a processing result, and report the log information and transmission packet corresponding to the processing result to the log service node corresponding to the target service node. The log service node then reports the log information and transmission packet to the log collection service, enabling the log collection service to determine the error log information and the corresponding call chain order based on the global identifier in the transmission packet. Since the global identifier is obtained by the gateway from the log collection service in real time, the gateway can automatically add the packet containing the global identifier to the external request and distribute the external request carrying the packet to the target service node. The target service node then processes the external request, obtains the processing result, and reports the log information and packet corresponding to the processing result to the log service node corresponding to the target service node. This allows the log collection service to automatically determine the error log information and the corresponding call chain order based on the global identifier in the packet. This enables precise tracking of the microservice call order, automatic discovery, reporting, and processing of error logs, and effectively improves the efficiency of error log analysis.

[0043] In one exemplary embodiment, the method further includes:

[0044] If an external request is received through the gateway and the external request is invalid, determine whether the client identifier corresponding to the external request is in the priority queue.

[0045] If the client identifier corresponding to an external request is not in the priority queue, the external request is ignored.

[0046] Specifically, we will illustrate this method by taking its application to a microservice cluster as an example. Figure 3As shown, a microservice cluster can receive external requests from different clients through a gateway (Nginx) and verify the validity of these requests. For example, when an external request A arrives, the Nginx gateway first checks if request A is valid. If request A is invalid, the gateway then checks if the client A (client identifier or user identifier) ​​to which request A belongs is in a priority queue. If it is in the priority queue, the gateway directly reports the invalidity error message of request A to the log collection service, LogCollection. Since the error message carries relatively little information at this stage, LogCollection can decide whether to persist the error message or output it to the dashboard for visualization based on the specific request content. If the client identifier A corresponding to request A is not in the priority queue, the gateway can ignore request A without processing it. This allows for rapid location of the error point based on the call chain and error log information, enabling efficient development and maintenance while significantly reducing manpower and time costs.

[0047] In one exemplary embodiment, the method further includes:

[0048] If the client identifier corresponding to the external request is in the priority queue, the error information corresponding to the external request is reported to the log collection service through the gateway, so that the log collection service can determine whether to persist or visualize the request based on the request content corresponding to the external request.

[0049] Specifically, we will illustrate this method by taking its application to a microservice cluster as an example. Figure 3As shown, a microservice cluster can receive external requests from different clients through a gateway (Nginx) and verify the validity of these requests. For example, when an external request A arrives, the Nginx gateway first checks if request A is valid. If request A is invalid, the gateway then checks if the client A (client identifier or user identifier) ​​belongs to request A is in a priority queue. If it is, the gateway directly reports the invalidity error message to the log collection service, LogCollection. Since the error message carries relatively little information, LogCollection can determine whether to persist the error message or display it on a dashboard for visualization based on the specific request content. This allows LogServer to output the most accurate error logs through analysis of the aggregated logs. Furthermore, configuration can enable dashboard display and database persistence, significantly improving error troubleshooting capabilities, increasing development and operational efficiency, reducing manpower and time costs, and effectively enhancing the efficiency of error log analysis.

[0050] In one exemplary embodiment, after obtaining the global identifier from the log collection service via the gateway, the method further includes:

[0051] Add the global identifier to the delivery packet;

[0052] The process of obtaining a processing result from the external request includes:

[0053] When processing external requests and needing to call other service nodes, obtain the request arrival time of the external request, the forwarding time of the external request, and its own node information;

[0054] The request arrival time, forwarding time, and node information are added to the delivery packet to obtain an updated delivery packet. The external request carrying the updated delivery packet is then forwarded to other service nodes for processing to obtain the processing result.

[0055] Specifically, such as Figure 3As shown, the microservice cluster can add a Context packet containing the global identifier Sequence1 to the external request A through the gateway Nginx, and distribute the external request A carrying the Context packet to the target service node Servers Cluster1, so that the target service node Servers Cluster1 can process the external request A, obtain the processing result A, and report the log information A corresponding to the processing result A and the Context packet to the LogServer corresponding to the target service node Servers Cluster1. In this scenario, if node A in the target service node Servers Cluster1 processes external request A and needs to call other service nodes (e.g., node B in servers Cluster2), node A in Servers Cluster1 can obtain the arrival time of external request A, the forwarding time of the external request, and its own node information (i.e., node A). It then adds the arrival time, forwarding time, and node information (i.e., node A) to the transmission packet Context, obtaining an updated transmission packet Context1. This updated transmission packet Context1 is then forwarded to node B in Servers Cluster1 for processing to obtain the result B. The result is then returned to node A in the target service node Servers Cluster1. In other words, the call order between the internal service nodes of external request A is node A → node B. Therefore, by adopting a context-based call chain tracing mechanism, the call order of microservices can be accurately tracked, enabling automatic discovery, reporting, and processing of error logs, thereby effectively improving the efficiency of error log analysis.

[0056] In an exemplary embodiment, the processing result includes a successful processing result and a failed processing result; the step of reporting the log information corresponding to the processing result and the transmission packet to the log service node corresponding to the target service node includes:

[0057] When the processing result is successful, the log information corresponding to the successful processing result is reported to the log service node corresponding to the target service node.

[0058] When the processing result is a failure, the error log information and update delivery packet corresponding to the failure result are reported to the log service node corresponding to the target service node. The update delivery packet includes a global identifier, request arrival time, forwarding time and node information.

[0059] Specifically, such as Figure 3As shown, the microservice cluster can add a Context packet containing the global identifier Sequence1 to the external request A through the gateway Nginx, and distribute the external request A carrying the Context packet to the target service node Servers Cluster1, so that the target service node Servers Cluster1 can process the external request A, obtain the processing result A, and report the log information A corresponding to the processing result A and the Context packet to the LogServer corresponding to the target service node Servers Cluster1. When the processing result A is successful, the target service node Servers Cluster1 reports the log information A and the delivery packet Context (or update delivery packet Context) corresponding to the successful processing result to the corresponding log service node LogServer. This reports the success information corresponding to the external request A, and the delivery packet Context only needs to contain the Sequence1 information. When the processing result A is a failure, the target service node Servers Cluster1 reports the error log information and the update delivery packet Context (or delivery packet Context) corresponding to the failure to the corresponding log service node LogServer. The update delivery packet Context includes a global identifier, request arrival time, forwarding time, and node information, so that the subsequent log collection service, LogCollection, can use the global identifier Sequence1 in the update delivery packet to find error log information with the same global identifier Sequence1 from all reported log information, and determine the call chain order corresponding to the error log information based on the request arrival time, forwarding time, and node information in the update delivery packet Context. Therefore, by adopting a context-based call chain tracing mechanism, it is possible to accurately track the call order of microservices, realize the automatic discovery, reporting and processing of error logs, and thus effectively improve the analysis efficiency of error logs.

[0060] In an exemplary embodiment, the step of determining error log information and the call chain order corresponding to the error log information based on a global identifier in the transmission packet includes:

[0061] Based on the global identifier in the update delivery packet, obtain error log information with the same global identifier;

[0062] Based on the request arrival time, forwarding time, and node information in the update packet, determine the call chain order corresponding to the error log information;

[0063] The method also includes: persistently storing or visually displaying error log information and call chain order.

[0064] Specifically, such as Figure 3 The LogCollection service shown can search for error logs with the same global identifier Sequence1 from all reported log information based on the global identifier Sequence1 in the update delivery package Context. It then determines the call chain order corresponding to the error log information based on the request arrival time, forwarding time, and node information in the update delivery package Context. For example, assuming the call order between the internal service nodes of external request A is node A → node B → node C, the LogCollection service, based on the global identifier Sequence1 in the update delivery package Context, searches for error logs with the same global identifier Sequence1 from all reported log information. It then determines the call chain order corresponding to the error log information as A → node B → node C based on the request arrival time, forwarding time, and node information in the update delivery package Context. This information is then output to the WebDashboard for visualization or persisted to the database, thus visualizing the call chain between microservices, enabling precise tracking, facilitating process tracing and problem troubleshooting, and making error location more accurate and intuitive.

[0065] In an exemplary embodiment, after determining the error log information and the corresponding call chain order based on the global identifier in the transmission packet, the method further includes:

[0066] Release the global identifier in the delivery packet so that the global identifier can be used to add to other external requests.

[0067] Specifically, such as Figure 3The LogCollection service shown in the diagram searches for error logs with the same global identifier Sequence1 from all reported log information based on the global identifier Sequence1 in the update delivery package Context. It then determines the call chain order corresponding to the error log information based on the request arrival time, forwarding time, and node information in the update delivery package Context. For example, assuming the call order between the internal service nodes of external request A is node A → node B → node C, the LogCollection service searches for error logs with the same global identifier Sequence1 from all reported log information, and determines the call chain order corresponding to the error log information as A → node B → node C. After outputting this information to the WebDashboard for visualization or persisting it to the database, the LogCollection service automatically releases the global identifier Sequence1 from the update delivery package Context so that it can be used to add other external requests. In other words, the LogCollection service releases the Sequence1 and reuses it for other requests. This allows for the visualization of call chains and automatic analysis of error logs, enabling frontline operations and maintenance personnel to conduct initial troubleshooting before development teams intervene, effectively reducing the workload of development teams and improving operational efficiency.

[0068] In one exemplary embodiment, the method provided in this application can be applied to the analysis of error logs in various scenarios. The following description uses an application to a microservice cluster as an example to illustrate the error log analysis method provided in this application.

[0069] The problems and drawbacks of the traditional approach include:

[0070] The internet is currently witnessing a vigorous technological revolution. With the increasing maturity of cloud-native technologies and the continuous emergence of related tools, more and more companies are embracing the cloud-native era, and the vision of "everything on the cloud" is slowly becoming a reality. Since its inception, Docker, leveraging the advantages of cloud services and the continuous improvement of container cluster deployment tools, has shone brightly in microservices and has been accepted by the vast majority of companies. However, while enjoying the convenience brought by microservices, problems have also arisen. The numerous services lead to long and complex call chains. Once an error occurs in a module, it is necessary to continuously trace upwards based on the error, a cumbersome and time-consuming process. If the error code is not written in a standardized manner, it may sometimes be impossible to find the root cause from the error, causing great trouble for development and operation and maintenance work.

[0071] Therefore, to address the aforementioned issues, this application proposes a context-based automatic error log analysis method, similar to Go's Context mechanism. This method continuously tracks and traces the call results along the call chain using Context, enabling automatic error log discovery, reporting, and processing. Finally, the error reporting information from the entire call chain is aggregated and integrated for display on a WebDashboard or for database persistence.

[0072] Among them, Nginx gateway: Nginx acts as a gateway, responsible for filtering and forwarding incoming requests. It can be combined with other gateway tools as needed to achieve richer and more complex gateway functions, such as multi-gateway integration, etc. For the sake of brevity, it will not be elaborated here.

[0073] LogCollection is a log collection and analysis service. It is responsible for receiving error log information sent from all LogServers, integrating and summarizing the information, and displaying the final results through a WebDashboard or persisting them.

[0074] Server Clusters: Represents a cluster of microservices on individual nodes, responsible for the actual business processing. It continuously calls other microservices based on actual requests until the entire request is processed. When an error occurs, it reports the error information to the LogServer service within the cluster, which then analyzes, integrates, and further reports the logs.

[0075] LogServer: Responsible for collecting error log information sent from various microservices within the cluster. It needs to identify and filter the error information corresponding to different requests, and then integrate and report these error messages to the LogServer or LogCollection in the upper-level call chain.

[0076] The purpose of this application in proposing an automatic error log analysis method based on context is:

[0077] 1. Accurately track the call chain between microservices: Visualize the call chain between microservices, accurately track it, and facilitate process tracking and problem troubleshooting.

[0078] 2. Reduce troubleshooting costs: Through a visualized call chain and automatic error log analysis mechanism, frontline operations and maintenance personnel can initiate troubleshooting, followed by R&D intervention. This effectively reduces the workload of R&D and improves operational efficiency.

[0079] 3. Enhance real-time monitoring of individual microservices: A visual dashboard allows for precise identification of specific call chains where problems have occurred within a microservice, going beyond simply macro-level data statistics. This further reduces the granularity of error display, making error localization more accurate and intuitive.

[0080] like Figure 3 As shown in the figure, the specific implementation process of the automatic error log analysis method based on context proposed in this application is as follows:

[0081] 1. When an external request arrives, the Nginx gateway performs a check. If the request is invalid, it checks if the user to which the connection belongs is in the priority queue. If it is in the priority queue, the error message is reported to the LogCollection. At this stage, the error message carries relatively little information; the LogCollection decides whether to persist it or output it to the Dashboard based on the specific request content. Otherwise, the invalid request is ignored, and no error is reported to the LogCollection. If the request is valid, Nginx requests a globally unique Sequence from the LogCollection to uniquely identify the current request and adds it to the Context. Context information is added to the current request, including things like the request arrival time, forwarding time, and information about the last processed node. After the Context information is added, step 2 is executed.

[0082] 2. At this point, the request will be handled by one of the services in the microservice cluster. If the current service can process the request content without errors, it returns the request result, ends the current request process, and reports a success message to the LogServer corresponding to the current cluster. At this time, the Context only needs to contain the Sequence information, and the LogServer continues to report until it reaches the LogCollection, which releases the Sequence and reuses it for other requests. If processing fails, the error message and the corresponding Context are reported to the LogServer corresponding to the current cluster. The LogServer integrates the error information and sends it to the upper-level LogServer or LogCollection (if there is no upper-level LogServer). When the LogCollection finishes processing, it releases the current Sequence. If the current service needs to further call services in other clusters to complete the processing of the request, it updates the Context and appends the time when the current node forwarded the request and the service node information. Repeat step 2.

[0083] 3. LogCollection first divides all received error messages according to the Sequence; then, for each identical message in the Sequence, LogCollection summarizes and organizes complete error log information with the correct call chain order according to the call and forwarding time and service node information, and outputs it to the WebDashboard or performs database persistence operations.

[0084] By following the steps above, developers and operations personnel will be able to handle numerous microservice errors with ease, quickly locating the error point directly based on the call chain and error log information. This will enable high-speed and efficient development and operations work, greatly reducing manpower and time costs.

[0085] The beneficial effects of the technical solution in this application include:

[0086] 1. A context-based call chain tracing mechanism is proposed to accurately track the call order of microservices.

[0087] 2. By analyzing the aggregated error logs, LogServer can output the most accurate error logs, and can be configured to display them on a large screen and persist them to the database.

[0088] 3. Development and operations personnel can perform precise error troubleshooting based on accurate error logs, which greatly improves error troubleshooting capabilities, increases development and operations efficiency, and reduces manpower and time costs.

[0089] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0090] Based on the same inventive concept, this application also provides an error log analysis apparatus for implementing the error log analysis method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more error log analysis apparatus embodiments provided below can be found in the limitations of the error log analysis method described above, and will not be repeated here.

[0091] In one exemplary embodiment, such as Figure 4 As shown, an error log analysis device is provided, including: an acquisition module 402, an addition module 404, a distribution module 406, and a reporting module 408, wherein:

[0092] The acquisition module 402 is used to acquire a global identifier from the log collection service through the gateway when an external request is received through the gateway and the external request is a legitimate request.

[0093] Add module 404, used to add a delivery packet containing the global identifier to the external request through the gateway.

[0094] The distribution module 406 is used to distribute the external request carrying the transmission packet to the target service node, so that the target service node can process the external request, obtain the processing result, and report the log information corresponding to the processing result and the transmission packet to the log service node corresponding to the target service node.

[0095] The reporting module 408 is used to report the log information and the transmission packet to the log collection service through the log service node, so that the log collection service can determine the error log information and the call chain order corresponding to the error log information based on the global identifier in the transmission packet.

[0096] In one embodiment, the apparatus further includes: a judging module, configured to judge whether the client identifier corresponding to the external request is in a priority queue when an external request is received through the gateway and the external request is an invalid request; and an ignoring module, configured to ignore the external request when the client identifier corresponding to the external request is not in the priority queue.

[0097] In one embodiment, the reporting module is further configured to report the error information corresponding to the external request to the log collection service through the gateway when the client identifier corresponding to the external request is in the priority queue, so that the log collection service can determine whether to perform persistent storage or visualization display based on the request content corresponding to the external request.

[0098] In one embodiment, the adding module is further configured to add the global identifier to the delivery packet; the obtaining module is further configured to, when processing the external request and needing to call other service nodes, obtain the request arrival time of the external request, the forwarding time of the external request, and its own node information; the adding module is further configured to add the request arrival time, the forwarding time, and the node information to the delivery packet to obtain an updated delivery packet, and forward the external request carrying the updated delivery packet to the other service nodes for processing to obtain the processing result.

[0099] In one embodiment, the processing result includes a successful processing result and a failed processing result; the reporting module is further configured to report the log information corresponding to the successful processing result to the log service node corresponding to the target service node when the processing result is a successful processing result; and to report the error log information corresponding to the failed processing result and the update delivery packet to the log service node corresponding to the target service node when the processing result is a failed processing result; wherein, the update delivery packet includes the global identifier, the request arrival time, the forwarding time, and the node information.

[0100] In one embodiment, the acquisition module is further configured to acquire error log information having the same global identifier based on the global identifier in the update delivery packet; the device further includes: a determination module, configured to determine the call chain order corresponding to the error log information based on the request arrival time, the forwarding time, and the node information in the update delivery packet; and a processing module, configured to persistently store or visualize the error log information and the call chain order.

[0101] The modules in the aforementioned error log analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0102] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores error log analysis data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an error log analysis method.

[0103] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0104] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0105] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0106] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0107] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0108] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0109] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0110] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for analyzing error logs, characterized in that, The method includes: If an external request is received through the gateway and the external request is invalid, determine whether the client identifier corresponding to the external request is in the priority queue. If the client identifier corresponding to the external request is in the priority queue, the error information corresponding to the external request is reported to the log collection service through the gateway, so that the log collection service can determine whether to perform persistent storage or visualization display based on the request content corresponding to the external request. If an external request is received through the gateway and the external request is a valid request, the global identifier is obtained from the log collection service through the gateway. The gateway adds a packet containing the global identifier to the external request, and distributes the external request carrying the packet to the target service node. The target service node processes the external request, obtains a processing result, and reports the log information corresponding to the processing result and the packet to the corresponding log service node. The target service node and the log service node are associated; the log service node is one of the service nodes in the target service node, which is a microservice cluster. The log service node is used to collect error log information sent by various microservices within the target service node. The process of reporting the log information and the delivery packet to the log service node of the upper-level call chain via the log service node includes: if the processing result is successful, reporting success information and the delivery packet to the log service node of the upper-level call chain, wherein the delivery packet only needs to contain the global identifier; if the processing result is unsuccessful, reporting error information and the delivery packet to the log service node of the upper-level call chain, wherein the delivery packet contains the global identifier, request arrival time, forwarding time, and node information; if there is no log service node of the upper-level call chain, then reporting the log information and the delivery packet to the log collection service, so that the log collection service can search for error log information with the same global identifier from all reported log information based on the global identifier in the delivery packet, and determine the error log information and the call chain order corresponding to the error log information according to the request arrival time, forwarding time, and node information in the delivery packet; the log collection service is a service for log collection and analysis. The global identifier in the delivery packet is released through the log collection service so that the global identifier can be used to add to other external requests.

2. The method according to claim 1, characterized in that, The method further includes: If the client identifier corresponding to the external request is not in the priority queue, the external request is ignored.

3. The method according to claim 1, characterized in that, After obtaining the global identifier from the log collection service through the gateway, the method further includes: Add the global identifier to the delivery packet; The process of obtaining a processing result from the external request includes: When processing the external request and needing to call other service nodes, obtain the request arrival time of the external request, the forwarding time of the external request, and its own node information; The request arrival time, the forwarding time, and the node information are added to the transmission packet to obtain an update transmission packet. The external request carrying the update transmission packet is then forwarded to the other service nodes for processing to obtain the processing result.

4. The method according to claim 3, characterized in that, The processing results include successful processing results and unsuccessful processing results; The step of reporting the log information corresponding to the processing result and the transmission packet to the log service node corresponding to the target service node includes: When the processing result is a successful result, the log information corresponding to the successful processing result is reported to the log service node corresponding to the target service node. When the processing result is a processing failure, the error log information corresponding to the processing failure and the update delivery packet are reported to the log service node corresponding to the target service node; wherein, the update delivery packet includes the global identifier, the request arrival time, the forwarding time and the node information.

5. The method according to claim 4, characterized in that, The step of determining the error log information and the call chain order corresponding to the error log information based on the global identifier in the transmission packet includes: Based on the global identifier in the update delivery packet, obtain error log information that has the same global identifier; Based on the request arrival time, the forwarding time, and the node information in the update delivery packet, determine the call chain order corresponding to the error log information; The method further includes: persistently storing or visually displaying the error log information and the call chain sequence.

6. An error log analysis device, characterized in that, The device includes: The judgment module is used to determine whether the client identifier corresponding to the external request is in the priority queue when an external request is received through the gateway and the external request is an invalid request. The reporting module is used to report the error information corresponding to the external request to the log collection service through the gateway when the client identifier corresponding to the external request is in the priority queue, so that the log collection service can determine whether to perform persistent storage or visualization display based on the request content corresponding to the external request. The acquisition module is used to obtain a global identifier from the log collection service through the gateway when an external request is received through the gateway and the external request is a legitimate request; An add module is used to add a packet containing the global identifier to the external request through the gateway; A distribution module is used to distribute the external request carrying the transmission packet to a target service node, so that the target service node processes the external request, obtains a processing result, and reports the log information corresponding to the processing result and the transmission packet to the log service node corresponding to the target service node. The target service node and the log service node are associated, and the log service node is one of the service nodes in the target service node, which is a microservice cluster. The log service node is a service node used to collect error log information sent by each microservice within the target service node. The reporting module is further configured to report the log information and the delivery packet to the log service node of the upper-level call chain through the log service node, including: if the processing result is a successful result, reporting success information and the delivery packet to the log service node of the upper-level call chain, wherein the delivery packet only needs to contain the global identifier; if the processing result is a failed result, reporting error information and the delivery packet to the log service node of the upper-level call chain, wherein the delivery packet contains the global identifier, request arrival time, forwarding time, and node information; if there is no log service node of the upper-level call chain, then reporting the log information and the delivery packet to the log collection service, so that the log collection service, based on the global identifier in the delivery packet, searches for error log information with the same global identifier from all reported log information, and determines the error log information and the call chain order corresponding to the error log information according to the request arrival time, forwarding time, and node information in the delivery packet; the log collection service is a service for log collection and analysis; the global identifier in the delivery packet is released through the log collection service so that the global identifier can be used to add to other external requests.

7. The apparatus according to claim 6, characterized in that, The device further includes: An ignore module is used to ignore an external request if the client identifier corresponding to the external request is not in the priority queue.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images