A database-based web service log analysis and error trace method, device and system

Abstract

This application discloses a database-based method, apparatus, and system for web service log analysis and error tracing, comprising: acquiring runtime exception logs containing thread ID, timestamp, and call stack, and interface access logs containing interface name, thread ID, interface entry timestamp, and response time; accurately matching the two types of log records based on the condition that the thread IDs are the same and the runtime exception timestamp is within the window formed by the interface entry timestamp and the interface entry timestamp plus the response time; associating the matched interface with the responsible developer based on a pre-defined interface-developer mapping relationship; and finally pushing the error details and developer information to the corresponding terminal. This application utilizes the thread ID + time window matching mechanism and responsibility mapping to automate the entire process from error discovery and responsibility determination to dispatch, solving the problems of difficult error location, unclear responsibility attribution, and low dispatch efficiency in large-scale development teams, and significantly shortening the problem resolution cycle.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a database-based method, apparatus, and system for web service log analysis and error tracing. Background Technology

[0002] With the widespread use of web applications and the increasing complexity of systems, locating errors or anomalies during operation has become extremely challenging. Currently, problem localization relies heavily on information from error logs, runtime logs, API logs, and other logs. While mainstream web frameworks have mature logging modules—for example, Tomcat in the Java domain provides rich logging resources that can be easily configured, including: catalina.log (containing logs of Tomcat server startup, shutdown, and operation), localhost.log (containing logs specific to a particular host), manager.log (containing TomcatManager application logs), host-manager.log (containing Tomcat Host Manager application logs), and localhost_access_log (containing access logs for each HTTP request)—these framework-provided logging modules generally suffer from a drawback: they lack developer information.

[0003] This deficiency has vastly different impacts on teams of different sizes. For small development teams, the framework's built-in logging module is usually sufficient to locate problems because the operations or testing personnel who discover the problem communicate smoothly with the developers who resolve it, and they might even be the same person. However, in larger development teams, the situation is entirely different. The person who discovers the problem (e.g., testers) and the person capable of resolving it (e.g., developers) are often not the same person, and may even belong to different departments and be strangers to each other. This decouples the problem's "discoverer" from the "solver," and also the "discovery phase" from the "resolution phase," urgently requiring the introduction of a "problem attribution and assignment phase" to connect the two. Therefore, for larger development teams, after discovering a problem, determining its attribution and assigning it to a developer for resolution becomes a significant challenge. The core technical issue is that in large-scale development teams, there is a lack of an automated mapping mechanism between error logs and specific development responsibilities. This makes it difficult to quickly locate the specific interface, code module, and corresponding developer after an error is discovered, thus prolonging the problem-solving cycle and increasing manual communication costs.

[0004] Existing technical solutions primarily rely on manual log sifting, experience-based error identification, and manual notification of developers. This process is inefficient, error-prone, and unsuitable for the complex call chains inherent in microservices and distributed architectures. This application aims to propose a systematic automated solution to address these challenges. Summary of the Invention

[0005] This application proposes a database-based method, apparatus, and system for web service log analysis and error tracing. It aims to address the technical problems in large-scale web service development teams where the lack of an automated mapping mechanism between error logs and specific development responsibilities leads to difficulties in quickly locating the responsible interface and developer after an error is discovered. This results in long error localization cycles, high manual communication costs, and low efficiency in assigning responsibility and distributing issues. This application achieves the following by constructing a log analysis system independent of the original web service: structured log collection and storage; database-based correlation matching; automated issue distribution and display; and system independence and portability.

[0006] In a first aspect, embodiments of this application provide a database-based method for web service log analysis and error tracing, comprising the following steps:

[0007] Obtain runtime exception logs and interface access logs. The runtime exception logs shall at least contain thread ID, timestamp, and call stack, and the interface access logs shall at least contain interface name, thread ID, interface entry timestamp, and interface response time.

[0008] Each error record in the runtime exception log is matched with a record in the interface access log. The matching conditions are: the thread IDs of the two records are the same, and the timestamp of the runtime exception log record is within a window starting from the interface entry timestamp of the interface access log record and lasting for the interface response time.

[0009] Based on the pre-established interface-developer mapping relationship, the interface corresponding to the successfully matched interface access record is mapped to the developer information responsible for the development;

[0010] Based on the mapping result, an error notification containing error details and developer information is pushed to the terminal associated with the developer.

[0011] In one embodiment, the matching is implemented based on relational database association queries, treating the runtime exception log and the interface access log as two separate database tables, and performing table association queries based on thread ID and time window conditions.

[0012] In one embodiment, prior to the above step, the method further includes: during the log configuration phase, mandating that the Web service framework output the runtime exception log and interface access log in a predetermined format, and ensuring that the thread ID in the interface access log and the thread ID in the runtime exception log originate from the same thread pool.

[0013] In one embodiment, the push notification is implemented by calling the application programming interface (API) of an external system, including an interface of an enterprise instant messaging tool, a ticketing system, or an email system, to convert error notifications into to-do tasks or instant messages.

[0014] In one embodiment, the step further includes: generating a visual statistical report based on the matched and mapped error records, the report including at least the distribution of errors by time, interface module, developer, and average resolution time.

[0015] In one embodiment, the matching is performed using a real-time stream processing framework. By maintaining a sliding time window mapping relationship between thread IDs and recent interface access records, second-level matching and tracing can be achieved when errors occur.

[0016] In one embodiment, under a distributed service architecture, the runtime exception log and interface access log also contain a globally unique request tracing ID; the matching is based on the request tracing ID to replace or assist the thread ID, thereby achieving end-to-end error tracing across services and processes.

[0017] In one embodiment, the final step also includes the following steps: based on historical matching and push data, intelligently assessing the urgency and scope of the error using a machine learning model, and dynamically adjusting the priority of the push and the recipients accordingly.

[0018] Secondly, embodiments of this application also provide a database-based Web service log analysis and error tracing apparatus for implementing the method described in the first aspect, comprising:

[0019] The log acquisition module is used to acquire runtime exception logs and interface access logs;

[0020] The matching engine module is used to match the error records in the runtime exception log with the records in the interface access log. The matching conditions are that the thread IDs are the same and the timestamps fall within the time window determined based on the interface entry timestamp and the response time.

[0021] The mapping module is used to map successfully matched interfaces to developers based on the interface-developer mapping relationship;

[0022] The push execution module is used to push notifications containing error and developer information to the terminal associated with the developer.

[0023] Thirdly, embodiments of this application also provide a database-based Web service log analysis and error tracing system, including:

[0024] One or more application servers are used to run web services and generate runtime exception logs and interface access logs;

[0025] A database server is used to store the logs and the interface-developer mapping relationship;

[0026] On an analysis server independent of the application server, the log analysis and error tracing device as described in the second aspect embodiment is deployed, or on an analysis server independent of the application server, the method as described in the first aspect embodiment is executed;

[0027] At least one developer terminal is used to receive and process error push notifications from the device.

[0028] In one embodiment, the analysis server further includes a report generation module, used to generate a visual statistical dashboard based on the data in the database server and display it through a management terminal.

[0029] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0030] Fifthly, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described in the first aspect.

[0031] In a sixth aspect, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0032] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0033] The method, apparatus, and system proposed in this application, through a creative precise matching mechanism based on "thread ID + time window" and responsibility mapping technology, achieve automated association and assignment of error logs with development responsibilities. This solves the problem of difficulty in quickly assigning responsibility for errors in large-scale teams, transforming the previously lengthy process relying on manual communication and experience-based judgment into an efficient, accurate, and automated technical process, thus shortening the problem-solving cycle. Specifically, this application has the following advantages:

[0034] This system boasts strong portability. It does not require being placed in the same codebase as the original web service, nor does it need to use the same programming language. Once developed and tested on one web service project, this system can be ported to other web projects with minimal code modifications.

[0035] It is non-intrusive to the original web service. The "log analysis" task is independent of the web service and can be started by another process or deployed on another server. Therefore, it will not affect the performance, reliability, or availability of the original web service. Its operation will not consume the resources of the original web service, and even if it fails, it will not affect the normal operation of the original web service.

[0036] Shorten the problem-solving cycle. This system can automatically identify problems, determine their attribution, and push them to the corresponding developers for resolution, thus shortening the problem-solving process cycle.

[0037] Automation saves manpower. This system is highly automated, and (after development and debugging) it can save some of the manpower previously used for "checking logs and locating problems".

[0038] The final results are highly readable. This application's "log analysis" system aggregates related data from multiple logs into summary tables organized by "issue," providing clear and concise results. Furthermore, displaying these results in report or page format facilitates better overall project management and team control for project managers and other team members. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a flowchart illustrating an embodiment of the method described in this application;

[0041] Figure 2 This is a flowchart of another embodiment of the method in this application, which includes a log configuration phase;

[0042] Figure 3 This is a flowchart of another embodiment of the method of this application, which includes a statistical report generation step;

[0043] Figure 4 This is a flowchart of another embodiment of the method of this application, which includes a real-time stream processing matching step;

[0044] Figure 5 This is a schematic diagram of a log analysis and problem discovery system;

[0045] Figure 6This is a structural block diagram of an embodiment of the device described in this application;

[0046] Figure 7 This is a structural block diagram of an embodiment of the system described in this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] The core innovations of this application lie in the following two aspects.

[0049] Based on a precise matching mechanism of "thread ID + time window", this approach creatively leverages the characteristic of the same thread processing the same request within a short period during web service runtime. By associating "thread ID" with a time window defined by "timestamp + response time", it precisely matches seemingly independent runtime exception logs with interface access logs, establishing a direct link between "error" and "the request that caused the error".

[0050] Responsibility mapping and automated assignment. By introducing a static configuration layer called "Interface-Developer Mapping Table", the technical interface matching results are transformed into management responsibility attribution and automatically pushed to specific developers. This achieves full-process automation from "error discovery" to "problem assignment", decoupling the problem discoverer from the solver.

[0051] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0052] Example 1: Core Method Flow

[0053] Figure 1 The flowchart below illustrates an embodiment of the method described in this application, detailing the core four-step method flow: log acquisition → matching → mapping → push.

[0054] Step 110: Obtain runtime exception logs and interface access logs.

[0055] This corresponds to "log configuration" and subsequent data collection. The runtime exception log must contain at least the thread ID, timestamp, and call stack. The call stack is the execution path information when the program encounters an error, which is crucial for locating the root cause of the problem. The interface access log must contain at least the interface name, thread ID, interface entry timestamp, and interface response time. The methods for "acquiring" this data are varied, such as reading from the file system, pulling from a network interface, or subscribing from a message broker. To ensure the effectiveness of subsequent steps, the runtime exception log and interface access log originate from a specifically configured web service. Specifically, during the log configuration phase, two types of logs must be configured: first, runtime exception logs, which can be output at the exception handling location through an exception capture mechanism (or error detection in the code). The error log must contain the following information: call stack, timestamp, error reason, and thread ID. Second, interface access logs, which can be formatted using the framework's built-in logging module (or a custom logging module called at the interface's entry and exit points). The interface access log must include the following information: interface name, thread ID that processed the interface request, interface access timestamp (entry point), and interface response time (usually provided by the framework; if you write your own, use the difference between the exit and entry timestamps). Configure the logs, specifying the formatted output to ensure critical information is recorded. This configuration step can be further understood as: during the log configuration phase, forcing the web service framework to output the runtime exception logs and interface access logs in a predetermined format, and ensuring that the thread IDs in the interface access logs and runtime exception logs originate from the same thread pool. This is the foundation for ensuring the matching logic holds true.

[0056] Step 120: Match each error record in the runtime exception log with the records in the interface access log.

[0057] This step implements "log analysis," which involves connecting log information through a "log analysis" task to ultimately compile an "error information summary table." The specific matching conditions are: the two records have the same thread ID, and the timestamp of the runtime exception log record falls within a window starting from the interface entry timestamp of the interface access log record and lasting for the interface response time. The key to matching lies in two fields: thread ID and timestamp. In the matching method of this application, the timestamp serves to limit the time range. Considering that the runtime exception log and the interface access log are generated on the same server (even in a distributed environment, logs on the same server use the same system time), if a record from the runtime exception log can match a record from the interface access log, their timestamps should be close (but not identical). Theoretically, their error should be less than the "interface response time" (the interface response time is recorded in the interface access log). For example, an erroneous interface access request occurred on "2025-1-1 12:00:00:000," and the interface response time was 800ms. Theoretically, the error occurred between "2025-1-1 12:00:00:000" and "2025-1-1 12:00:00:800". Therefore, by limiting the time stamp range using the timestamp and the interface response time, and then precisely matching it with the thread ID, the corresponding interface access record in the error log can be found. This process completes the technical correlation: through the above matching mechanism, the specific interface request that caused the error is automatically identified, solving the problem of "unknown error source".

[0058] Step 130: Based on the pre-established interface-developer mapping relationship, map the interface corresponding to the successfully matched interface access record to the developer information responsible for its development.

[0059] This involves entering information into the "Interface Developer Information Table." This table records the correspondence between interfaces and developers. It includes at least the interface name, developer / department, the controller / service file to which the interface entry belongs, and the name of the function entry point for processing the interface. Entering interface information can be a mandatory requirement for code submission (if a new interface is not registered in the "Interface Developer Information Table," it will be rejected); alternatively, it can be entered uniformly through subsequent collection or developer submission. After successful matching in step 120, a joint query with the "Interface Developer Information Table" can piece together complete information for each error record (interface information, including the interface entry; error information, including the complete call stack; interface developer / department). This step completes the responsibility association: through the mapping table, interfaces are automatically associated with specific developers or teams, resolving the problem of "unclear responsibility attribution."

[0060] Step 140: Based on the mapping result, push the error notification containing error details and developer information to the terminal associated with the developer.

[0061] This step implements "problem push." ​​Specifically, this push can be achieved by calling the application programming interfaces (APIs) of external systems, including enterprise instant messaging tools, ticketing systems, or email systems, to transform error notifications into to-do tasks or instant messages. For example, when summarizing data in the "Error Information Summary Table," errors are categorized into different "functional modules" based on the call stack, and problems under different "functional modules" are pushed to different groups. The members of these groups are the developers of the different "functional modules." Alternatively, it can be interconnected with the company's "company management system" and "bug feedback system," dispatching problems to developers for resolution and displaying a "to-do" status under the developer's employee account. Problem resolution can also be linked to employee performance and workload. This step automates the process: through system push, problems are automatically delivered to responsible personnel, solving the problem of "inefficient dispatch processes."

[0062] Example 2: Detailed method including database implementation and complete workflow

[0063] Figure 2 The flowchart of another embodiment of the method in this application includes a log configuration phase, illustrating a complete closed loop from log configuration, collection, storage, matching, mapping, push to statistics.

[0064] Step 210: During the log configuration phase, the Web service framework is required to output the runtime exception logs and interface access logs in a predetermined format.

[0065] This step refines the preparation phase of step 110. This constitutes a prerequisite for the method, ensuring that the acquired logs possess the necessary key fields. Simultaneously, to ensure the log analysis task can successfully retrieve logs, the "log analysis" task is generally a separate process from the original web service. From a feasibility standpoint, it can also be deployed on a separate server. Therefore, during deployment, attention must be paid to firewall, log file permissions, and log file visibility configurations to ensure that the "log analysis" task can retrieve all log files generated by the original web service.

[0066] Step 220: Obtain and preprocess the execution log file.

[0067] This step involves log collection and storage. The "Log Analysis" task iterates through the log file line by line, splitting or merging single or multiple lines of data. Important fields are extracted and stored in the database. Several points should be noted during this process:

[0068] (1) Avoid long-term single-time occupation and avoid blocking and waiting of the original web service due to reading logs.

[0069] (2) Multiple lines of data in the log file do not necessarily mean multiple error records (e.g., call stacks. Dozens of call stack information should belong to one error record). Multiple lines of data belonging to the same record should be merged.

[0070] (3) Before entering data into the database, check for duplicate data to avoid duplicate entries.

[0071] The reading, splitting, and insertion of log files can be further optimized: the volume of interface access logs is typically between hundreds of thousands and tens of millions of records. To facilitate subsequent analysis and matching, some auxiliary fields need to be generated before insertion (e.g., generating discrete values ​​for "response speed" based on continuous values ​​of "interface response time," and generating an integer type for "thread ID" based on the string type of "thread ID," for subsequent analysis and index creation during insertion). During insertion, frequent database access should be avoided. Insertion statements for 100,000 records can be concatenated into a single insertion statement. This way, only one database access is needed for every 100,000 records inserted.

[0072] Step 230: Match the error records in the runtime exception log with the records in the interface access log, wherein the matching is implemented based on the relational database association query.

[0073] This step refines step 120. Specifically, the runtime exception logs and interface access logs already stored in the database are treated as two separate database tables. SQL statements are written to perform table join queries using thread IDs and time window conditions (i.e., the matching conditions defined in step 120). This leverages the characteristics of relational databases in "matching and joining data" to link information from multiple related (but requiring some form of matching) log files together, allowing for the analysis and extraction of more intuitive information.

[0074] The timing for starting the "log analysis" task can be: (1) during idle time or at a set time. In this case, during the process of reading logs and storing them in the database, it is necessary to record the position of the "last read" to avoid repeated reading and unnecessary overhead. At the same time, deduplication should be done when storing data (if the timestamp is highly accurate, the timestamp can be used as the basis for deduplication). (2) When a new log file is detected (if log sharding is configured). In this case, if the sharding settings of the interface access log and the sharding settings of the runtime exception are different, there may be a situation where "a new runtime exception log is generated, but no new interface access log is generated". Then, the newly added runtime exception log obviously cannot immediately find an interface access record that can be matched with it. In this case, two processes can be set up to complete the "log analysis" task together. One is used to "store the log file in the database"; the other is used to "traverse the error information summary table, logs that do not match the interface access record, and try to match them in the newly added data in the interface access record table". With this method, there will be a certain delay in matching, but in the end, each error message will definitely be matched with the interface access record, and they will correspond one-to-one.

[0075] Step 240 (corresponding to step 130): Map responsibility based on the interface-developer mapping relationship.

[0076] Step 250 (corresponding to step 140): Push the error notification to the terminal associated with the developer, wherein the push is implemented by calling the application programming interface of an external system.

[0077] Based on Example 1, Example 2 further specifies the matching database implementation method and the standardized output requirements of logs, thus constituting a specific and implementable solution.

[0078] Example 3

[0079] Figure 3 This is a flowchart of another embodiment of the method in this application, which includes a statistical report generation step. This embodiment, based on Embodiment 1 or Embodiment 2, adds a data summarization and display step, which can follow step 140 or step 250:

[0080] Step 310: Generate a visual statistical report based on the matched and mapped error records.

[0081] This is an implementation of the "visualization and statistics" process. The reports should at least include the distribution of errors by time, interface module, and developer, as well as the average resolution time. Furthermore, an intuitive and visually appealing front-end page can be developed to display recent (e.g., the past week) issues, providing both issue details and statistical data. This allows project managers and other team members to have a better grasp of the overall project. This also reflects the high readability of the final result presented in this application.

[0082] Example 4: Optimization Method Based on Real-Time Stream Processing

[0083] Figure 4 This is a flowchart of another embodiment of the method of this application, which includes a real-time stream processing matching step. This embodiment is a further optimization of the matching step (step 120), corresponding to optimization scheme one: real-time stream processing. The batch processing "matching" step is upgraded to real-time stream processing based on a sliding time window, which solves the matching delay problem.

[0084] Step 410: Obtain runtime exception logs and interface access log streams in real time.

[0085] Alternatively, in step 410, the runtime exception logs and interface access logs are derived from a log message stream persisted by a message middleware. Unlike batch file reading, this step is typically implemented by subscribing to log topics in a message queue (such as Kafka).

[0086] Step 420: The matching is performed using a real-time stream processing framework.

[0087] Existing solutions rely on timed or triggered batch processing, resulting in matching delays. To address this issue, this embodiment introduces a stream processing framework (such as Apache Flink or Kafka Streams) to achieve second-level matching and tracing of errors by maintaining a sliding time window mapping between thread IDs and recent interface access records. Specifically, in the stream processing engine's memory state, a sliding time window is maintained for each active thread ID (the window size can be set according to the maximum interface response time). This window continuously receives and retains interface access records processed by that thread within a recent period. When an exception log record is received, the engine immediately locates the corresponding sliding window based on its thread ID and quickly searches for records that meet the time window condition (error timestamp within the range of [interface entry timestamp + response time]) among all interface access records within the window, thus achieving millisecond-level matching. This method utilizes the time sliding window mechanism, combined with possible Bloom filters for rapid deduplication, significantly improving the system's real-time performance.

[0088] Example 5: Optimization Method for Adapting to Distributed Architecture and Integrated Intelligent Evaluation

[0089] This embodiment describes two further optimizations to Embodiment 1, designed to enhance architectural adaptability and processing intelligence. They can be applied independently or in combination to the foregoing embodiments.

[0090] Optimization Scheme 2: Matching Enhancement Based on Distributed Tracking

[0091] This optimization aims to address the issue of core matching logic (based on thread ID) failing in microservice or distributed architectures. Its core lies in enhancing step 120 (matching).

[0092] Step 510: In a distributed service architecture, configure and ensure that the runtime exception logs and interface access logs output by the Web service also contain a globally unique request trace ID (TraceID).

[0093] This step can be incorporated into step 110 / Preparation stage.

[0094] This is typically achieved by integrating an SDK with a distributed tracing system (such as Jaeger or SkyWalking), allowing logs across services in the same request chain to share the same TraceID.

[0095] In a distributed service architecture, the runtime exception logs and interface access logs also contain globally unique request trace IDs. This is to address the issue in microservice architectures where a request may span multiple services, and relying solely on thread IDs is insufficient for cross-service tracing. By integrating a distributed tracing system (such as Jaeger or SkyWalking) and using TraceID instead of thread ID as the association key, end-to-end error tracing is achieved.

[0096] Specifically, in the process of obtaining logs, it is necessary to ensure that a distributed tracing ID is injected into the logs.

[0097] Step 520: When matching the error records in the runtime exception log with the records in the interface access log, the matching is based on the request tracking ID, which replaces or assists the thread ID.

[0098] This step can replace or enhance the matching logic in step 120.

[0099] In the matching step, the matching is based on the request trace ID, replacing or supplementing the thread ID. This means that the matching logic can be adjusted to: prioritize using the request trace ID (TraceID) for accurate cross-service matching; if the request trace ID is missing, fall back to using the thread ID for intra-service matching. The applicability extends from single-service to microservice / distributed architectures. This technical implementation enables this solution to achieve end-to-end error tracing across services and processes, solving the cross-process tracing challenge.

[0100] The specific implementation includes two modes:

[0101] Alternative mode: In the matching criteria, "same request trace ID" completely replaces the "same thread ID" condition. The time window condition remains unchanged. This enables precise matching across services and processes.

[0102] Secondary mode: Prioritizes matching using the request tracing ID; if the request tracing ID is missing or invalid, it falls back to using the thread ID for in-service matching. This is a more compatible implementation.

[0103] Steps 510 and 520 achieve end-to-end error tracing across services and processes, extending the applicability of this method from single-machine / single-service systems to complex distributed systems. Subsequent steps 130 (mapping) and 140 (push) do not require modification.

[0104] Optimization Solution 3: Enhanced Push Notifications Based on Intelligent Evaluation

[0105] This optimization aims to address the issue of still requiring manual judgment of priority and impact scope after an error is pushed to the developer. Its core lies in enhancing step 140 (push notification) by adding a smart evaluation sub-step before the push.

[0106] Step 530: Based on historical matching and push data, intelligently assess the urgency and impact of the current erroneous record using a pre-trained machine learning model.

[0107] This step can be set after step 130 and before step 140 as a newly added intelligent evaluation step. After an error is pushed to developers, manual judgment of priority and impact scope is still required. To solve this problem, a machine learning model is introduced to automatically recommend error priorities and possible root cause modules based on features such as historical error data, call frequency, and interface importance. Specifically, a classification model (such as XGBoost) can be used to train a classification model on features of historical error records (such as error type, QPS of the interface where the error occurred, and importance of associated modules) to obtain an error classification model. Before or during the push, this model is invoked to evaluate new errors, automatically assigning priorities (such as P0-P3), and potentially dynamically adding relevant module responsible persons to be notified based on the impact scope analyzed by the call chain graph database. This achieves an evolution from "simple dispatch" to "intelligent operation," improving processing efficiency.

[0108] Specific sub-steps may include:

[0109] 530a. Feature extraction: Extract features from current error records, matched interface access records, and historical databases, such as error type, occurrence time, recent call frequency and error rate of the interface, business importance of the module to which the interface belongs, and upstream dependencies in the call chain.

[0110] 530b. Model Inference: Input feature vectors into a machine learning model (such as a classification model trained with XGBoost) and output the erroneous estimated urgency level (e.g., P0-P3) and a list of modules or teams that may be affected.

[0111] Step 540: Based on the intelligent evaluation results, dynamically adjust the priority of push notifications and the recipients.

[0112] This step enhances the push logic in step 140. Specifically, it includes:

[0113] Priority Adjustment: The urgency level output by the model will be used as the priority label for push messages. High-priority errors can be pushed immediately through more prominent channels (such as telephone, strong reminder IM), while low-priority errors can be included in the daily summary report.

[0114] Dynamic adjustment of receiving objects: In addition to the responsible developers identified in mapping step 130, error information can also be pushed to the relevant module leaders or teams that may be affected, based on the scope of impact obtained in step 530b, to facilitate collaborative investigation.

[0115] Through steps 530 and 540, the machine learning model intelligently classifies and recommends errors, achieving an evolution from "simple dispatch" to "intelligent operation," improving the efficiency and accuracy of problem handling. The logic of steps 110 (log acquisition), 120 (matching), and 130 (mapping) remains unchanged, but the data they generate is given higher decision-making value.

[0116] In one of the most complete embodiments, the method flow is based on four core steps (steps 110 to 140) and integrates various optimization and enhancement functions. Steps 110 to 150 of the embodiments of this application transform the originally lengthy process that relied on manual communication and experience judgment into an efficient, accurate, and automated technical process through technical association, responsibility and association, and process automation.

[0117] The other steps are concretized and expanded around the core chain described above. First, steps 210 (force log configuration) and 220 (collect and store data) are executed as preparation and data preprocessing for step 110 (obtain logs). Next, the core matching step is executed, where there are two possible implementation paths: one is to use step 230 (database-based relational query) to implement step 120; the other is to use the more real-time steps 410 (real-time log stream acquisition) and 420 (sliding window matching based on a real-time stream processing framework) to implement step 120. If the system is in a distributed architecture, steps 510 (injecting tracking IDs) and 520 (matching based on request tracking IDs) need to be incorporated into this matching stage to enhance or replace the original thread ID matching logic. After matching is completed, step 130 (responsibility mapping) is executed. Subsequently, an intelligent evaluation stage is introduced before push, namely steps 530 (intelligent evaluation based on machine learning models) and 540 (dynamically adjusting push strategy), and finally, error notifications are distributed by step 250 (pushing through external system interfaces) or step 140. After the entire process is completed, step 310 (generating a visual statistical report) can be performed to evaluate the effectiveness and gain insights into the project.

[0118] Therefore, steps 110-140 form the main framework, while steps 210-250 and 310 provide details on the basic implementation. Steps 410-420 provide performance optimization paths (Optimization Solution 1), steps 510-520 provide architectural adaptability extensions (Optimization Solution 2), and steps 530-540 provide an intelligent decision-making layer (Optimization Solution 3). These steps are interconnected and inclusive, together forming a complete, efficient, scalable, and intelligent solution for Web service log analysis and error tracing.

[0119] Example 6: Device Example

[0120] Figure 6 This is a structural block diagram of an embodiment of the device described in this application, illustrating a database-based Web service log analysis and error tracing device 600, used to implement the method described in any one of the first aspects of this application, including:

[0121] The log acquisition module 61 is used to acquire runtime exception logs and interface access logs. This module corresponds to step 110 in Embodiment 1 and the collection parts of steps 210 and 220 in Embodiment 2, and can be implemented through an Agent or a file listener.

[0122] The matching engine module 62 is used to match error records in the runtime exception log with records in the interface access log. The matching condition is that the thread IDs are the same and the timestamps fall within a time window determined based on the interface entry timestamp and response time. This module is the core, corresponding to step 120 in Embodiment 1 and step 230 in Embodiment 2. It can be implemented as a service for performing database association queries (corresponding to claim 2), or as a stream processing computing node (corresponding to claim 6 and step 420 in Embodiment 3), or as an engine that supports distributed tracking ID queries (corresponding to claim 7 and Embodiment 4).

[0123] The mapping module 63 is used to map successfully matched interfaces to developers based on the interface-developer mapping relationship. This module corresponds to step 130 in Embodiment 1 and step 240 in Embodiment 2, and is responsible for querying the "Interface Developer Information Table" to complete the responsibility mapping.

[0124] The push execution module 64 is used to push notifications containing error and developer information to the terminal associated with the developer. This module corresponds to step 140 in Embodiment 1 and step 250 in Embodiment 2, and integrates interfaces from external systems such as WeChat Work, DingTalk, and JIRA.

[0125] The device 600 can be deployed on an analysis server independent of the web service, demonstrating the advantages of the system being non-intrusive to the original web service and independently deployable.

[0126] Example 7: System Example

[0127] Figure 7 The diagram below shows an embodiment of the system described in this application. A database-based Web service log analysis and error tracing system 700 includes:

[0128] One or more application servers 71 are used to run web services and generate runtime exception logs and interface access logs. These servers are the producers of raw logs, corresponding to the various web services described in the background section.

[0129] Database server 72 is used to store the logs and interface-developer mapping relationships. Specifically, it may include: a log repository (storing structured runtime exception logs and interface access logs); a mapping relationship repository (storing interface developer information tables); an issue summary repository (storing matched error records, push status, and resolution status); and a statistics and analysis repository (storing aggregated report data), etc.

[0130] The analysis server 73 is equipped with the log analysis and error tracing device 600 as described in Embodiment Six, or is configured to execute the method. Functionally, this server can be divided into: a log acquisition server (implementing the log acquisition module 61); a log analysis server (implementing the matching engine module 62 and mapping module 63); a push and integration server (implementing the push execution module 64); a report and API server; and a scheduling and monitoring server. The analysis server 73 also includes a report generation module (not shown separately in the figure), used to generate a visual statistical dashboard based on the data in the database server and display it through the management terminal 75, which corresponds to step 260 in Embodiment Two.

[0131] At least one developer terminal 74 is used to receive and process error push notifications from the device. In addition, the system also includes a management terminal (for project managers to view statistical reports), an operations and maintenance terminal (for monitoring system operation status and configuring data collection strategies), and a testing terminal (for viewing interface error trends), collectively forming a terminal layer (categorized by operator).

[0132] Figure 5 This diagram illustrates a "Log Analysis and Problem Discovery System," clearly demonstrating the complete system concept and data flow from log generation and analysis to problem notification. The left side shows the web service (corresponding to the application server) that generates runtime exception logs and interface access logs; the middle shows the "log analysis" task or server (corresponding to the analysis server and device) that performs log analysis and error tracing; the right side shows the destination of the analysis results, including push notifications to developers, report generation, and possible system management interfaces. This diagram macroscopically confirms... Figure 7The feasibility of the system architecture and workflow shown intuitively demonstrate the advantages of this application, such as automation, saving manpower, and good readability of the final results.

[0133] The system works as follows:

[0134] Log generation and collection: Web services output standardized logs during operation. Log collection servers collect logs through agents or file listening, parse them, and write them to a log repository.

[0135] Log matching and responsibility mapping: The log analysis server reads data from the log repository periodically or in real time, associates abnormal records with interface records through matching algorithms, queries the mapping relationship library to obtain developer information, and writes it into the issue summary library.

[0136] Issue Push and Handling: The push server retrieves unprocessed errors from the issue summary database and pushes them to the developer's terminal (such as DingTalk messages or JIRA tasks) via an integration interface. After the developer completes the handling, they update the status on their terminal, and the status is synchronized back to the issue summary database.

[0137] Data visualization and statistics: The report server extracts data from the issue summary database and the statistics database, and provides chart-based displays to the management terminal and the test terminal through the API, supporting multi-dimensional queries and exports.

[0138] System scheduling and monitoring: The scheduling server manages the task coordination and resource allocation of each server, and the monitoring server collects system indicators and sends alarms to the operation and maintenance terminal when anomalies occur.

[0139] It should be noted that the execution entities of each step in the log analysis and error tracing method provided in the embodiments of this application can be the same physical or logical device, or they can be executed collaboratively by different devices in a distributed manner. For example, the log collection and storage step (such as step 220) can be executed by an agent deployed on the application server side, while the log matching and analysis step (such as steps 120, 230, 420) and the mapping step (step 130) can be executed by an independent analysis server; furthermore, the error push step (steps 140, 250) can be executed by a dedicated integrated push server. As another example, in the real-time stream processing optimization embodiment, step 410 (real-time acquisition) and step 420 (stream matching) may be completed collaboratively by different computing nodes in the stream processing cluster. This flexibility of execution entities and distributed deployment capability is precisely the embodiment of the non-intrusive and highly scalable design advantages of this solution.

[0140] Other implementation methods

[0141] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] Therefore, this application also proposes a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the methods described in any embodiment of this application.

[0143] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] Furthermore, this application also proposes an electronic device (or computing device) including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any embodiment of this application.

[0147] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, a network interface, and memory. Memory may include non-persistent storage in computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media. Computer-readable media includes both permanent and non-persistent, removable and non-removable media; information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of a program, or other data.

[0148] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

Claims

1. A database-based method for web service log analysis and error tracing, characterized in that, Includes the following steps: Obtain runtime exception logs and interface access logs. The runtime exception logs shall at least contain thread ID, timestamp, and call stack, and the interface access logs shall at least contain interface name, thread ID, interface entry timestamp, and interface response time. Each error record in the runtime exception log is matched with a record in the interface access log. The matching conditions are: the thread IDs of the two records are the same, and the timestamp of the runtime exception log record is within a window starting from the interface entry timestamp of the interface access log record and lasting for the interface response time. Based on the pre-established interface-developer mapping relationship, the interface corresponding to the successfully matched interface access record is mapped to the developer information responsible for the development; Based on the mapping result, an error notification containing error details and developer information is pushed to the terminal associated with the developer.

2. The method according to claim 1, characterized in that, The matching is implemented based on relational database association queries, treating the runtime exception log and interface access log as two separate database tables, and performing table association queries based on thread ID and time window conditions.

3. The method according to claim 2, characterized in that, Prior to the above steps, the following steps are also included: during the log configuration phase, the Web service framework is required to output the runtime exception log and interface access log in a predetermined format, and it is ensured that the thread ID in the interface access log and the thread ID in the runtime exception log originate from the same thread pool.

4. The method according to claim 1, characterized in that, The push notification is implemented by calling the application programming interfaces (APIs) of external systems, including enterprise instant messaging tools, ticketing systems, or email systems, to convert error notifications into to-do tasks or instant messages.

5. The method according to claim 1, characterized in that, Following the aforementioned steps, the process further includes generating a visual statistical report based on the matched and mapped error records. The report includes at least the distribution of errors by time, interface module, and developer, as well as the average resolution time.

6. The method according to claim 1, characterized in that, The matching is performed using a real-time stream processing framework. By maintaining a sliding time window mapping relationship between thread IDs and recent interface access records, it achieves second-level matching and source tracing when errors occur.

7. The method according to claim 1, characterized in that, In a distributed service architecture, the runtime exception logs and interface access logs also contain globally unique request tracing IDs; the matching is based on the request tracing IDs to replace or assist the thread IDs, thereby achieving end-to-end error tracing across services and processes.

8. The method according to claim 1, characterized in that, Finally, the following steps are also included: based on historical matching and push data, a machine learning model is used to intelligently assess the urgency and scope of the error, and the priority and recipients of the push are dynamically adjusted accordingly.

9. A database-based Web service log analysis and error tracing device, used to implement the method described in any one of claims 1 to 8, characterized in that, include: The log acquisition module is used to acquire runtime exception logs and interface access logs; The matching engine module is used to match the error records in the runtime exception log with the records in the interface access log. The matching conditions are that the thread IDs are the same and the timestamps fall within the time window determined based on the interface entry timestamp and the response time. The mapping module is used to map successfully matched interfaces to developers based on the interface-developer mapping relationship; The push execution module is used to push notifications containing error and developer information to the terminal associated with the developer.

10. A database-based Web service log analysis and error tracing system, characterized in that, include: One or more application servers are used to run web services and generate runtime exception logs and interface access logs; A database server is used to store the logs and the interface-developer mapping relationship; On an analysis server independent of the application server, deploy the log analysis and error tracing device as described in claim 9, or perform the method as described in any one of claims 1 to 8; At least one developer terminal is used to receive and process error push notifications from the device.

Images