API log collection method, error monitoring method, checking method and system
By tagging API logs to establish business requirements and link relationships, the problems of scattered logs and lack of correlation are solved, enabling efficient error monitoring and rapid problem localization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI DAZHIHUI CAIHUI DATA TECH CO LTD
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-21
AI Technical Summary
In a microservice architecture, API logs are scattered across different nodes and lack an effective correlation mechanism, making it difficult to quickly locate the root cause of problems and monitor complex business requests.
By tagging newly added event logs during the API log collection process, a unique event identifier, input data, output data, and timestamp are generated and stored in a distributed database. This establishes a correspondence between logs and business requirements and links, and builds a parent-child event relationship.
It enables efficient and accurate association of API logs, reduces computing resource consumption, improves the efficiency of error detection and problem localization, and enhances the operational efficiency of technical personnel.
Smart Images
Figure CN121901076A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of log collection technology, and in particular relates to an API log collection method, error monitoring method, viewing method and system. Background Technology
[0002] In existing distributed system monitoring technologies, the collection of API (Application Programming Interface) logs has the following problems: ① Log distribution problem: In a microservice architecture, a business request often involves multiple service calls, and the logs of these services are distributed across different nodes. Nodes are used to execute corresponding operations, and node logs are generated when the corresponding operation is executed within a node.
[0003] ② Lack of correlation: There is a lack of effective correlation mechanisms. When multiple services are performed on the same transaction (e.g., multiple changes to the same table are required), the nodes containing related service logs are independent. Therefore, it is impossible to effectively and accurately obtain all service logs and nodes related to the current transaction at once. This makes it difficult to quickly troubleshoot the problematic node when a problem occurs in the current transaction. Furthermore, interference exists between multiple services, so even if the service logs of each node have timestamps, it is difficult to locate the root cause node when a problem occurs in the current transaction.
[0004] This also makes it impossible to fully monitor complex business requests involving interactions between multiple things (such as updating the main table, which is built based on multiple sub-tables) by checking the logs.
[0005] This application content
[0006] The purpose of this application is to overcome the shortcomings of the existing technology and provide an API log collection method that enables the logs to have a unique correspondence with business requirements, the chain to which they belong, and the location of the embedded points in the chain during the API log collection process. This lays the foundation for efficient and accurate troubleshooting of embedded points when discovering problems in the business later.
[0007] To achieve the above objectives, this application adopts the following technical solution: An API log collection method includes the following steps: S1, determine the backend based on business needs; S2 retrieves newly added event logs from the backend in real time, tags the new event logs, and stores them in a distributed database.
[0008] Preferably, tagging newly added event logs also includes the following: recording the tagged event log as a tagging log; the event log includes the event identifier of the current event, input data, executed operation, output data, and the timestamp of generating the current event log; determining whether the current event has a parent event based on the content of the current event log; if it does not, generating a new global link identifier and binding it together with the business requirements with the current event log to obtain the tagging log of the current event; if determining that the current event has a parent event based on the content of the current event log, binding the parent event's event identifier, the parent event's global link identifier, and the parent event's business requirements with the current event log to obtain the tagging log of the current event.
[0009] Preferably, it also includes the following: if it is found that the operation fails after n consecutive operations within a certain tracking point, an error log is generated and recorded in the error flow table. The error log contains information about the newly added tagging log corresponding to the tracking point.
[0010] This application also provides a method for monitoring API log errors, characterized by including the following: When a new error log is added to the error flow table, the corresponding newly tagged log is retrieved from the distributed database based on the new error log; then, the remaining tagged logs on the relevant link are retrieved based on the corresponding newly tagged log, and an alert is sent to the technical personnel; the error flow table is obtained using an API log collection method as described above.
[0011] This application also provides a method for viewing API logs, including the following steps: Step 1: Based on the log viewing needs of technical personnel, obtain tagged logs that are directly related to these needs; the tagged logs are obtained using one of the API log collection methods described above. Step 2: After retrieving all the tagging logs on the link where the directly related tagging logs are located, the logs are fed back to the technical personnel in the form of the link.
[0012] Preferably, step 1 also includes the following: the log viewing requirement is to view all tagged logs containing a certain key field, or to view all tagged logs corresponding to a certain business requirement, or all logs within a certain time period; using keyword search or time period filtering, several directly related tagged logs for the current log viewing requirement are obtained.
[0013] Preferably, step 2 also includes the following: based on the parent tracking point identifier and the parent tracking point link global identifier recorded in a directly related tagging log, determine the parent tracking point of the tracking point corresponding to the current directly related tagging log, and sequentially construct the tracking point link where the tracking point corresponding to the current directly related tagging log is located, retrieve the tagging logs corresponding to each tracking point in the tracking point link, sort them according to the link form, and feed them back to the technical personnel.
[0014] This application also provides an API log collection system, including: a backend selection module, a log tagging module, and a distributed database; the backend selection module is used to determine the backend according to business requirements; the log tagging module is used to obtain newly added event logs from the backend in real time, tag the newly added event logs, and store them in the distributed database; each module is programmed or configured to perform the steps of an API log collection method as described above.
[0015] This application also provides a computer-readable storage medium, characterized in that: the computer-readable storage medium stores a computer program that is programmed or configured to perform an API log collection method as described above.
[0016] This application also provides a computer program product, including a computer program / instructions, characterized in that: when the computer program / instructions are executed by a processor, they implement the steps of the API log collection method described above.
[0017] The beneficial effects of this application are as follows: (1) The API log collection method of this application, in the process of collecting new event logs, marks the new event logs so that the marked logs stored in the distributed database have a unique correspondence with the business requirements, the link to which they belong, and the location of the event in the link. This makes the marked logs related to each other as belonging to the same business requirement or even the same link, laying the foundation for efficient and accurate error event investigation when the front end discovers problematic business.
[0018] (2) If the new logs in each tracking point are monitored separately in real time, the computational resources will be huge. However, the API log collection method of this application only tags the logs when new logs are added to the tracking points to obtain tagged logs. This not only eliminates the need to monitor each tracking point separately in real time and reduces the computational resources, but also ensures that each tagged log is still stored independently in the distributed database, and the tagged logs will not significantly increase the storage resources of the distributed database.
[0019] (3) An API log error monitoring method of this application can promptly report errors to technical personnel and simultaneously retrieve all the tagging logs on the link where the error reporting point is located. This allows technical personnel to directly view the logs of all upstream points that may cause the error when they receive an active error report, making it easier for them to find the cause of the active error report in the shortest possible time.
[0020] (4) When technicians find errors at the front end, they will check the event logs at the back end. However, the event logs are stored separately in the distributed database, which is a fragmented state. This application does not need to change the storage form of the logs in the distributed database, making the method of this application more universal.
[0021] (5) In the API log viewing method of this application, in order to facilitate technicians to better discover the cause of the error, we first determine several directly related labeled logs according to the log viewing needs of technicians, and then use the parent tracking point information in these directly related labeled logs to construct the corresponding tracking point chain. After retrieving the labeled logs corresponding to each tracking point in the tracking point chain, we sort them according to the chain form and feed them back to the technicians. In this way, the tracking point logs that technicians can directly see contain the time sequence of the operation and the data flow relationship, and what they can see is a log presented in the form of a complete chain, which makes it easier for technicians to find the cause of the error in the shortest time.
[0022] (6) In the prior art, technicians need to spend a lot of time setting log filtering conditions in order to avoid missing error-related logs, and it is impossible to guarantee that key logs will not be missed. Furthermore, the filtered logs are too scattered and lack correlation. Even if they have timestamps, they are still a mess for technicians and cannot quickly locate the problem node. The API log viewing method of this application is highly flexible. No matter what simple standard is used to filter the logs to be viewed, it can obtain the logs on the complete tracking link and the operation sequence and data flow relationship between tracking points based on the tagged logs. Subsequently, based on the link, the problem node can be easily located by tracing the upstream and downstream tracking points, which greatly reduces the time spent by technicians in setting log filtering standards before viewing logs.
[0023] (7) The API log viewing method of this application greatly improves the operation and maintenance efficiency of technical personnel. Attached Figure Description
[0024] Figure 1 This is a flowchart of an API log collection method according to this application. Detailed Implementation
[0025] To make the technical solution of this application clearer and more explicit, the application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Solutions derived by those skilled in the art through equivalent substitution and conventional reasoning of the technical features of the technical solution of this application without creative effort all fall within the protection scope of this application.
[0026] This application involves collecting node logs from the backend. The frontend primarily consists of the web page. Users send requests from the frontend to the backend, which are then transformed into several business requirements from the backend. After fulfilling these requirements and obtaining the results, the backend sends feedback back to the frontend. This is the backend responding to a frontend request.
[0027] Example 1
[0028] This application provides a method for collecting API logs, such as... Figure 1 As shown, for a specific business requirement, the following steps are included: S1, determine the backend based on business needs.
[0029] The choice of backend is determined based on the type of business requirements. There are two types of backends: Provider and Consumer. The Provider is used to handle business requirements such as message queues, caching, file storage, and database table changes. The Consumer is used to consume Kafka messages and services or data provided by other Providers.
[0030] S2 retrieves newly added event logs from the backend in real time, tags the new event logs, and stores them in a distributed database.
[0031] The backend contains many nodes. The nodes that implement a certain business requirement (the nodes will perform corresponding operations) are instrumented points. Therefore, the nodes that add new logs in the backend must be instrumented points (if a node implements a certain business requirement, it means that the corresponding node operation is executed in the node, and a new log will be generated in that node).
[0032] In S2, tagging newly added event logs also includes the following: Record the tagged event logs as tagging logs; The event tracking log contains the event tracking identifier of the current event tracking point, input data, executed operations, output data, and the timestamp of the event tracking log being generated. Based on the content of the current event tracking log, it is determined whether the current event tracking point has a parent event tracking point. If it does not, a new global link identifier is generated and bound to the current event tracking log along with the business requirements to obtain the tagging log of the current event tracking point. If it is determined from the content of the current event tracking log that the current event tracking point has a parent event tracking point, the parent event tracking identifier, the parent event tracking point's global link identifier, and the parent event tracking point's business requirements are bound to the current event tracking log to obtain the tagging log of the current event tracking point.
[0033] By knowing the tracking points in a link and the parent-child relationships between tracking points, the operation sequence within the tracking points used to implement business requirements and the direction of data transmission can be clearly defined.
[0034] The parent tracking point's tracking point identifier, the parent tracking point's global tracking point identifier, and the parent tracking point's business requirements are obtained from the latest tagging log of the parent tracking point in the distributed database. The tracking point identifier remains unchanged across different tagging logs generated at different times for the same tracking point, but the global tracking point identifier is different for each.
[0035] What needs to be explained here is: Because the event log contains the event identifier of the current event, input data, executed operations, output data, and the timestamp of the event log, it is possible to determine whether the current event log has a parent event based on its content.
[0036] In S2, a business operation may involve multiple global identifiers for all its event logs; however, an event log has only one global identifier. For event logs contained within the same global identifier: the input data recorded in the event log of a child event log originates from the output data recorded in the event log of its parent event log; the operation recorded in the event log of a child event log must occur after the operation recorded in the event log of its parent event log; and the operations recorded in the event logs of the parent and child event logs must occur sequentially.
[0037] The relationship between parent and child tracking points is determined by the sequential occurrence of operations within each tracking point. The time interval between the generation of tracking logs for parent and child tracking points is extremely short. Even for two identical business processes implemented sequentially, with the same parent and child tracking points, their global link identifiers will differ. Therefore, for a child tracking point, the parent tracking point identifier, the parent tracking point's global link identifier, and the parent tracking point's business requirement from the latest tagging log in the distributed database will be used to tag the tracking logs of that child tracking point. Tracking point A acts as the parent tracking point of tracking point C in the current business process, but in the previous business process, tracking point A acted as the parent tracking point of tracking point B. Therefore, the distributed database will store tagging logs of tracking point A at different times, but the tracking point identifiers in these logs will be the same, while the global link identifiers will be different.
[0038] For example, the operation in the tracking point is: "Read the main table information" in tracking point D, and then tracking point D sends the read main table information into the tracking point; the operation in tracking point E is "Get the sub-table related to the main table information", and then tracking point E sends the related sub-table into tracking point F; the operation in tracking point F is "Write the related sub-table into a certain system".
[0039] The API log collection method of this application tags the newly added event logs during the collection process, so that the tagged logs stored in the distributed database have a unique correspondence with the business requirements, the link they belong to, and the location of the event logs in the link. This makes the tagged logs related to the same business requirement or even the same link, laying the foundation for efficient and accurate debugging of error event logs when the front end discovers problems in the business.
[0040] Optionally, if it is found that all operations fail after n consecutive attempts within a certain tracking point, an error log is generated and recorded in the error flow table. The error log contains information about the new tagging log corresponding to that tracking point.
[0041] In this embodiment, n=3.
[0042] This self-reporting process means that when a certain tracking point generates an error log, even if there are downstream tracking points that need to be operated on later, they will no longer be operated, that is, the operation of the link where the error tracking point is located will be terminated.
[0043] This application achieves cross-service interconnection through link pass-through, that is, through the Kafka message header and error flow table, the instrumentation identifier, the link global identifier, etc. are passed through.
[0044] This application implements log extraction using the log facade pattern of the SLF4J framework.
[0045] The specific method for storing tagging logs within the distributed database can be defined by the user. It can be stored by field. In this embodiment, it is stored by time, with the tagging logs of each day stored together.
[0046] If we were to monitor the new log entries at each tracking point individually in real time, the computational resource consumption would be enormous. However, this application only tags the log entries when they are added at a specific tracking point to obtain tagged logs. This eliminates the need for real-time individual monitoring of each tracking point, reducing the computational resource consumption. Furthermore, it ensures that each tagged log is still stored independently in the distributed database, and the tagged logs do not significantly increase the storage resource consumption of the distributed database.
[0047] Example 2
[0048] This application provides a method for monitoring API log errors, including the following: When a new error log is added to the error flow table, the corresponding newly added tagged log is retrieved from the distributed database. Based on this newly added tagged log, the remaining tagged logs on the link to which the newly added tagged log belongs are retrieved, and an alert is sent to technical personnel. The error flow table is obtained during the API log collection process using an API log collection method as described in Example 1.
[0049] This application provides an API log error monitoring method that can promptly report errors to technical personnel and simultaneously retrieve all tagged logs on the link where the error-reporting point is located. This allows technical personnel to directly view the logs of all upstream points that may have caused the error upon receiving the proactive error report, enabling them to discover the cause of the proactive error report in the shortest possible time.
[0050] Example 3
[0051] This application provides a method for viewing API logs, including the following: Step 1: Based on the log viewing needs of technical personnel, filter out the tagged logs that are directly related to the log viewing needs; the tagged logs are obtained by using an API log collection method as described in Example 1.
[0052] Step 2: After retrieving all the tagging logs on the link where the directly related tagging logs are located, the logs are fed back to the technical staff for review in the form of the link.
[0053] Step 1 also includes the following: Technical personnel's log viewing needs might include viewing logs containing a specific key field, viewing all tagged logs corresponding to a certain business requirement, or viewing all logs within a certain time period.
[0054] By using keyword search or time period filtering, you can obtain several directly related tagged logs for the current log viewing needs.
[0055] Step 2 also includes the following: Based on the parent tracking point's tracking point identifier and the parent tracking point's link global identifier recorded in a directly related tracking point log, the parent tracking point of the tracking point corresponding to the current directly related tracking point log is determined. After constructing the tracking point link where the tracking point corresponding to the current directly related tracking point log is located, the tracking point logs corresponding to each tracking point in the tracking point link are retrieved, sorted according to the link form, and fed back to the technical personnel for review.
[0056] When technicians discover errors on the front end, they check the event tracking logs on the back end. However, these logs are stored separately in the distributed database, resulting in a fragmented state. This application does not require changing the storage format of the logs in the distributed database, making the method more versatile. To help technicians better identify the cause of errors, we first identify several directly related tagged logs based on their log viewing needs. Then, using the parent event tracking information in these directly related tagged logs, we construct the corresponding event tracking chain. After retrieving the tagged logs corresponding to each event tracking point in the chain, we sort them according to the chain structure and provide feedback to the technicians. In this way, the event tracking logs that technicians can directly see contain the time sequence of operations and the data flow relationship, and they can see logs presented as a complete chain. This makes it easier for technicians to find the cause of errors in the shortest possible time (because some errors are caused by errors occurring at the root event tracking point of a chain, but the logs viewed by technicians may only cover a certain time period and not include that root event tracking point).
[0057] Only when technicians need to view the logs will they build a complete tracking link based on the directly tagged logs, thus obtaining the upstream and downstream relationships between each tracking point; otherwise, the link-related information is simply hidden within the tagged logs.
[0058] In existing technologies, technicians need to spend a lot of time setting log filtering conditions to ensure that no error-related logs are missed, and it is impossible to guarantee that critical logs are not overlooked. Furthermore, the filtered logs are too scattered and lack correlation; even with timestamps, they are still a jumbled mess for technicians, making it impossible to quickly locate the problem node. The API log viewing method proposed in this application is highly flexible, and regardless of the simple criteria used to filter the logs to be viewed, it can obtain the logs along the complete tracking chain and the operation sequence and data flow relationships between tracking points based on the tagged logs. Subsequently, based on the chain, the problem node can be easily located by tracing upstream and downstream tracking points, significantly reducing the time spent by technicians in setting log filtering criteria before viewing logs.
[0059] Using the API log viewing method of this application to locate problem nodes, the time required is reduced to within 30 minutes compared to the existing technology which takes at least 2 hours, reducing the time by nearly 80% and significantly improving the operation and maintenance efficiency of technical personnel.
[0060] Example 4
[0061] This application also provides an API log collection system, including: The backend selection module, log tagging module, and distributed database; The backend selection module is used to determine the backend based on business requirements; The log tagging module is used to obtain newly added event logs from the backend in real time, tag the newly added event logs, and store them in a distributed database.
[0062] Each module is programmed or configured to perform the steps of an API log collection method as described in Example 1.
[0063] This application also provides a computer-readable storage medium storing a computer program programmed or configured to perform an API log collection method as described in Embodiment 1.
[0064] This application also provides a computer program product, including a computer program / instructions that are executed by a processor to implement the steps of an API log collection method as described in Embodiment 1.
[0065] The technologies, shapes, and structures not described in detail in this application are all well-known technologies. It should also be noted that the above are merely preferred embodiments created in this application and are not intended to limit the creation of this application. The components or steps in the embodiments of this application can be decomposed and / or recombined, and these decompositions and / or recombinations should be considered as equivalent solutions of this application and should all fall within the protection scope of this application.
Claims
1. A method for collecting API logs, characterized in that, Includes the following steps: S1, determine the backend based on business needs; S2 retrieves newly added event logs from the backend in real time, tags the new event logs, and stores them in a distributed database.
2. The API log collection method according to claim 1, characterized in that, Tagging newly added event logs also includes the following: recording the tagged event log as a tagged log; the event log contains the event identifier of the current event, input data, executed operation, output data, and the timestamp of the current event log being generated; Based on the content of the current event log, determine whether the current event has a parent event. If it does not, generate a new global link identifier and bind it together with the business requirements to the current event log to obtain the tagging log of the current event. If based on the content of the current event log, determine that the current event has a parent event, bind the parent event's event identifier, the parent event's global link identifier, and the parent event's business requirements to the current event log to obtain the tagging log of the current event.
3. The API log collection method according to claim 2, characterized in that, Also includes the following: If an operation fails after n consecutive attempts within a certain tracking point, an error log is generated and recorded in the error flow table. The error log contains information about the new tagging log corresponding to that tracking point.
4. A method for monitoring API log errors, characterized in that, Includes the following: When a new error log is added to the error flow table, the corresponding newly added tagged log is retrieved from the distributed database based on the newly added error log; then, the remaining tagged logs on the relevant link are retrieved based on the corresponding newly added tagged log, and an alarm is sent to the technical personnel; the error flow table is obtained using an API log collection method as described in any one of claims 1-3.
5. A method for viewing API logs, characterized in that, Includes the following steps: Step 1: Based on the log viewing needs of technical personnel, obtain the tagged logs that are directly related to the log viewing needs; the tagged logs are obtained using an API log collection method as described in any one of claims 1-3. Step 2: After retrieving all the tagging logs on the link where the directly related tagging logs are located, the logs are fed back to the technical personnel in the form of the link.
6. The API log viewing method according to claim 5, characterized in that, Step 1 also includes the following: the log viewing requirement is to view all tagged logs containing a certain key field, or to view all tagged logs corresponding to a certain business requirement, or all logs within a certain time period; using keyword search or time period filtering, obtain several directly related tagged logs for the current log viewing requirement.
7. The method for viewing API logs according to claim 5, characterized in that, Step 2 also includes the following: Based on the parent tracking point identifier and the parent tracking point link global identifier recorded in a directly related tagging log, determine the parent tracking point of the tracking point corresponding to the current directly related tagging log, and sequentially construct the tracking point link where the tracking point corresponding to the current directly related tagging log is located. Then, retrieve the tagging logs corresponding to each tracking point in the tracking point link, sort them according to the link form, and feed them back to the technical personnel.
8. An API log collection system, characterized in that, include: The system comprises a backend selection module, a log tagging module, and a distributed database. The backend selection module is used to determine the backend based on business requirements. The log tagging module is used to obtain newly added event logs from the backend in real time, tag the newly added event logs, and store them in the distributed database. Each module is programmed or configured to perform the steps of an API log collection method as described in any one of claims 1-3.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that is programmed or configured to perform an API log collection method as described in any one of claims 1-3.
10. A computer program product comprising a computer program / instructions, characterized in that: When the computer program / instruction is executed by the processor, it implements the steps of an API log collection method as described in any one of claims 1-3.