Automated defect localization test method and apparatus based on log analysis

Abstract

The application discloses an automatic defect positioning test method and device based on log analysis, first, log collection rules are configured and unified access specifications are established, while executing automatic test cases, test logs, system logs, interface logs, database logs and abnormal stack and other multi-source logs are collected in real time; after the original logs are cleaned, de-duplicated and structured converted, relying on an abnormal feature library, various defects are recognized through multi-dimensional matching of keywords, error codes, regulars, stack features and the like; then, in combination with service calling links, execution time sequences and module dependency relationships, defect positioning is completed and defect levels are automatically divided, finally, a standardized defect report is generated and is associated with test case execution results. The application can realize unattended intelligent defect diagnosis, has high positioning efficiency and low missed detection rate, is suitable for web system, micro service, interface and end-to-end test scenes, can effectively shorten defect positioning time and reduce manual debugging and communication costs.

Description

Technical Field

[0001] This invention relates to the field of software testing technology, and more specifically to an automated defect location testing method and apparatus based on log analysis. Background Technology

[0002] With the popularization of web systems, microservice architectures, and interface automation and end-to-end automated testing technologies, automated testing has become an important means of software quality control. At present, conventional automated testing can only determine whether the test case execution result is pass or fail, and cannot automatically identify defect types or trace the root cause of problems, which has obvious technical shortcomings.

[0003] When test cases fail, staff need to manually search and sort through various logs to troubleshoot the problem. On the one hand, the logs of the system under test are massive, and manually reviewing each log is time-consuming and laborious, significantly reducing the efficiency of test troubleshooting. On the other hand, the logs of microservices and distributed systems are scattered across multiple services and nodes, and various runtime logs, interface logs, and stack traces are fragmented, making it difficult to quickly connect the complete abnormal call chain, further increasing the difficulty of defect troubleshooting.

[0004] Meanwhile, traditional defect localization relies heavily on the practical experience and business familiarity of technical personnel, with a significant proportion of subjective judgment, making it prone to misjudgment and omission of defects, and compromising the accuracy of the localization results. Existing technologies also cannot streamline the entire process from test execution, anomaly capture, defect localization, to report output, making it difficult to achieve unattended intelligent defect diagnosis, resulting in high costs for software debugging and cross-functional communication.

[0005] In summary, to address issues such as low efficiency of manual log analysis, difficulty in connecting distributed logs, insufficient accuracy in defect location, and lack of full-process automation capabilities, the industry urgently needs to develop an automated testing solution based on log analysis that can complete anomaly identification and precise defect location in real time. Summary of the Invention

[0006] To address these issues, this invention provides an automated defect location testing method and apparatus based on log analysis. This method solves the problems of existing automated testing methods that can only determine the success or failure of test cases, require manual log retrieval which is inefficient, are difficult to connect distributed logs, rely on experience for defect location which is prone to misjudgment and omission, and cannot automate the entire process from test execution to defect reporting.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an automated defect location testing method based on log analysis, comprising:

[0008] S1. Configure log collection rules and establish a unified multi-source log access standard;

[0009] S2. Execute automated test cases and collect raw multi-source logs in real time;

[0010] S3. Perform standardized cleaning and structured transformation on the original multi-source logs to generate structured log data;

[0011] S4. Perform multi-dimensional matching on the structured log data based on the anomaly feature library to identify defect types and abnormal events;

[0012] S5. Based on the service call chain, log execution sequence relationship, and system module dependency path, locate the location of the defect, the root cause category, and the scope of impact;

[0013] S6. Integrate the defect identification and location results, generate a standardized defect location report, and associate the standardized defect location report with the execution results of the corresponding test cases.

[0014] As a preferred solution for automated defect location testing method based on log analysis, in S1, the log collection rules include the log path, log type, collection fields, and collection frequency of the system under test;

[0015] The log types include runtime logs, API logs, and error logs;

[0016] The collected fields include timestamp, traceId, service name, level, content, IP, and port;

[0017] The multi-source logs include system operation logs, interface request and response logs, database logs, microservice link logs, and error stack logs.

[0018] As a preferred solution for automated defect location testing methods based on log analysis, in S2, the original multi-source logs include test logs, application logs, interface logs, database logs, and exception stack logs.

[0019] As a preferred solution for automated defect location testing methods based on log analysis, in S3, the standardized cleaning includes deduplication of duplicate logs, filtering of noisy logs, completion of missing key fields, standardization of timestamp format, and cross-node link ID association mapping of the original logs;

[0020] The formula for unifying the timestamp format is:

[0021]

[0022]

[0023] In the formula, This is the original timestamp. This is a timestamp conversion function. For standard Unix timestamps, For time formatting functions, To format the time string;

[0024] The cross-node link ID association mapping matching formula is as follows:

[0025]

[0026] In the formula, Logs for different nodes. To extract logs The unique identifier of the link, To associate the matching results, This indicates that the link association was successful. This indicates that the link association failed;

[0027] The structured transformation process includes converting unstructured text logs into structured log data in JSON format, filtering out irrelevant logs, and retaining exception logs and main process logs.

[0028] As a preferred solution for automated defect location testing methods based on log analysis, in S4, the abnormal feature library includes error codes, abnormal keywords, regular expressions, abnormal types, return code rules, timeout thresholds, and business verification rules.

[0029] The multi-dimensional matching includes system error code matching, abnormal keyword matching, regular expression matching, abnormal stack feature matching, business return code matching, and interface timeout threshold judgment.

[0030] The formula for determining the interface timeout threshold is as follows:

[0031] ,

[0032] In the formula, The time consumed by the call, For the requested time, For response time, This is the timeout threshold. Marked as timeout. The timeout was detected. Determined as normal;

[0033] The formula for matching logs with anomaly features similarity is:

[0034]

[0035]

[0036] In the formula, This is the content of a single structured log entry. For keyword / exception stack / regular expression matching templates in the exception feature library, This is the set of overlapping characters between the log and the feature template. This is a function for calculating character length. The similarity between logs and anomaly features. For similarity threshold, As an anomaly marker, For the hit defect, No abnormalities were found.

[0037] The formula for matching the service return code is:

[0038]

[0039] In the formula, For real-time return codes, This is a set of exception return codes. Mark the return code as abnormal;

[0040] The defect types include null pointer exception, interface timeout exception, service internal error, business parameter exception, database operation exception, and business rule validation failure exception.

[0041] As a preferred solution for automated defect location testing methods based on log analysis, the anomaly feature library has a pre-set set of basic anomaly features covering general software systems, and also provides the function of user-defined business-specific anomaly features.

[0042] The abnormal feature library also has a self-learning capability, extracting new abnormal features based on historical defect location results and updating the abnormal feature library regularly.

[0043] The frequency formula for the anomaly feature library is:

[0044]

[0045] In the formula, Add new abnormal features to the candidates. This is a historical defect log dataset. Statistical characteristics In the dataset The number of times it appears in Features Frequency of occurrence;

[0046] The formula for updating the anomaly feature database with newly added anomaly features is as follows:

[0047]

[0048] In the formula, The threshold for feature admission frequency. This is the original feature set of the anomaly feature library; To add new feature markers, To be included in the feature library and updated.

[0049] As a preferred solution for automated defect location testing based on log analysis, the specific process of defect location in S5 is as follows: trace the complete distributed service call chain according to the traceId and spanId in the exception log, combine the timestamp execution sequence of the log with the predefined module dependency path diagram of the system, locate the service name, module name, interface address, function method, code level / system configuration item where the defect occurred, and reverse deduce the exception propagation path and the scope of affected business modules.

[0050] As a preferred solution for automated defect location testing methods based on log analysis, it also includes defect classification. After defect identification is completed, the defect score is calculated according to the comprehensive scoring formula, and the identified defects are automatically classified into four levels: blocking, severe, general, and warning. The corresponding processing priority and response time limit are marked for each level.

[0051] The comprehensive scoring formula is as follows:

[0052]

[0053] In the formula, Score based on the scope of business impact. Assign a score to the system crash risk. The degree of business disruption is scored. Score the user's perceived priority. The weights are fixed.

[0054] As a preferred solution for automated defect location testing methods based on log analysis, in S6, the standardized defect location report includes test case name, test execution time, test case execution result, defect unique identifier, defect level, defect type, location, related abnormal log fragments, abnormal propagation chain diagram, and suggested handling direction;

[0055] The formula for encoding the unique identifier of the defect is:

[0056]

[0057] In the formula, As a globally unique identifier for defects, Each test case has a unique identifier. To standardize Unix timestamps, For hash functions;

[0058] Once the standardized defect location report is generated, it is pushed to the defect management system and associated with the execution records and test screenshots of the corresponding test cases.

[0059] This invention also provides an automated defect location testing device based on log analysis, employing the aforementioned automated defect location testing method based on log analysis, comprising:

[0060] Configuration module: Used to configure log collection rules and establish a unified multi-source log access standard;

[0061] Log collection module: Used to execute automated test cases and collect raw multi-source logs in real time;

[0062] Log cleaning module: used to perform standardized cleaning and structured transformation on the original multi-source logs to generate structured log data;

[0063] Defect identification module: used to perform multi-dimensional matching on the structured log data based on the anomaly feature library to identify defect types and abnormal events;

[0064] Defect localization module: used to locate the location, root cause category, and scope of impact of defects based on service call chain, log execution sequence relationship, and system module dependency path;

[0065] Report generation module: Used to integrate defect identification and location results, generate standardized defect location reports, and associate the standardized defect location reports with the execution results of corresponding test cases.

[0066] The present invention has the following advantages:

[0067] First, this invention can eliminate the need for manual log retrieval and analysis, and automatically locate defects within seconds when automated testing fails, achieving unattended intelligent defect diagnosis and significantly reducing manual intervention.

[0068] Secondly, this invention enables unified collection and standardized processing of multi-source logs such as test logs, system logs, interface logs, database logs, and exception stacks. It is compatible with web systems, microservices, and distributed architectures, and can be applied to various scenarios such as interface automation and end-to-end testing, making it highly versatile.

[0069] Third, this invention relies on mechanisms such as anomaly feature database, keyword matching, and link tracing to identify defects. It has a high accuracy rate and a low false negative rate. At the same time, it supports complete backtracking of abnormal call links, which can accurately locate the defect location, root cause category, and scope of impact.

[0070] Fourth, this invention can automatically classify defects into four levels: blocking, severe, general, and warning. The defect classification and grading standards are clear, which facilitates the hierarchical management and handling of problems. It can automatically generate standardized defect location reports and associate them with test case execution results. The information is complete and traceability is convenient, further optimizing the logical management process of test problems.

[0071] Fifth, this invention automates the entire process of test execution, anomaly capture, defect location, and report output, effectively improving overall testing efficiency while significantly reducing the overall costs of system debugging and cross-functional communication. Attached Figure Description

[0072] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0073] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0074] Figure 1 This is a schematic diagram of the automated defect location testing method based on log analysis provided in Embodiment 1 of the present invention;

[0075] Figure 2 This is a technical roadmap for the automated defect location testing method based on log analysis provided in Embodiment 1 of the present invention;

[0076] Figure 3 This is a schematic diagram of the architecture of the automated defect location testing device based on log analysis provided in Embodiment 1 of the present invention. Detailed Implementation

[0077] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0078] Example 1

[0079] See Figure 1 and Figure 2 This invention provides an automated defect location testing method based on log analysis, comprising:

[0080] S1. Configure log collection rules and establish a unified multi-source log access standard;

[0081] Specifically, this step addresses the issue of inconsistent access to heterogeneous logs from multiple sources. Different systems and services exhibit significant differences in log output formats, storage paths, and field definitions. Without establishing unified standards beforehand, subsequent cleaning and analysis cannot be carried out. By standardizing collection rules, all systems under test are forced to output logs in a compliant format, laying the data foundation for subsequent fully automated processing.

[0082] S2. Execute automated test cases and collect raw multi-source logs in real time;

[0083] Specifically, this step synchronizes test execution and log collection, ensuring a one-to-one correspondence between log data and test case execution. Real-time collection avoids log loss or timing discrepancies, guaranteeing that complete contextual data at the time of defect occurrence is captured, thus resolving the issues of incomplete information and timing confusion caused by traditional post-event log collection.

[0084] S3. Perform standardized cleaning and structured transformation on the original multi-source logs to generate structured log data;

[0085] Specifically, raw logs are mostly unstructured text, containing a large amount of redundant, noisy, and missing information, making them unsuitable for direct machine analysis. This step cleanses and removes invalid data, completes missing information, and then converts the unstructured text into machine-readable structured data, such as JSON format, thus standardizing the log data and providing a computational data foundation for subsequent feature matching and defect localization.

[0086] S4. Perform multi-dimensional matching on the structured log data based on the anomaly feature library to identify defect types and abnormal events;

[0087] Specifically, this step uses feature matching to detect anomalies. The anomaly feature library pre-stores the feature patterns of various known defects. By matching the structured logs with the feature library in multiple dimensions, the abnormal information contained in the logs can be quickly identified and the defect type can be determined, replacing the process of manually searching for anomalies line by line.

[0088] S5. Based on the service call chain, log execution sequence relationship, and system module dependency path, locate the location of the defect, the root cause category, and the scope of impact;

[0089] Specifically, this step solves the problem of "finding the anomaly but not the root cause". A single anomaly log can only reflect the error of a certain node, while the call chain can trace the complete propagation path of the anomaly, and the timing relationship can determine the order in which the anomalies occurred. Then, the module dependency relationship can be used to analyze the scope of the anomaly's impact, ultimately achieving the leap from "discovering the anomaly" to "locating the root cause".

[0090] S6. Integrate the defect identification and location results, generate a standardized defect location report, and associate the standardized defect location report with the execution results of the corresponding test cases.

[0091] Specifically, this step standardizes the output and logical management of defect diagnosis results. By generating reports in a unified format, all key information about the defect can be presented; by associating the reports with test cases, a complete chain of "test case execution failure, defect location, and report traceability" can be achieved, facilitating testers to quickly reproduce problems and developers to accurately fix defects.

[0092] In one possible embodiment, in S1, the log collection rules include the log path, log type, collection fields, and collection frequency of the system under test;

[0093] Specifically, the log path specifies the storage location of the log file on the server, which is a prerequisite for the collector to find and read the log; the log type is used to distinguish logs for different purposes, which facilitates targeted processing later; the collection field defines the main information to be extracted and filters irrelevant fields to reduce the amount of data; the collection frequency controls the real-time nature of log collection and balances system performance and data timeliness.

[0094] The log types include runtime logs, API logs, and error logs;

[0095] Specifically, the operation log records the normal operating status of the system, used to reconstruct the business execution process; the interface log records the call requests and responses between services, and is the main data for analyzing the call chain of a distributed system; the error log specifically records the exceptions and error information thrown by the system, and is the main basis for defect identification. These three types of logs cover all dimensions of system operation information, and none can be omitted.

[0096] The collected fields include timestamp, traceId, service name, level, content, IP, and port;

[0097] Specifically, the timestamp is used to determine the precise time when the log was generated, which is the basis for time-series analysis; the traceId is a unique identifier of the distributed call chain, used to connect the logs of different service nodes; the service name, IP, and port are used to locate the specific node where the log was generated; the log level (INFO / WARN / ERROR) is used to quickly filter abnormal logs; the log content contains specific execution information and error details.

[0098] The multi-source logs include system operation logs, interface request and response logs, database logs, microservice link logs, and error stack logs.

[0099] Specifically, multi-source logs cover all levels of system operation. System operation logs reflect the operating system and application process status; interface logs reflect inter-service interactions; database logs reflect data operation anomalies; microservice chain logs reflect distributed call relationships; and error stack logs reflect code-level exception details. Comprehensive log collection ensures that the root cause of defects is not missed at any level.

[0100] In one possible embodiment, in S2, the original multi-source logs include test logs, application logs, interface logs, database logs, and exception stack logs.

[0101] Specifically, test logs record the execution process and results of automated test cases, used to associate defects with specific test cases; application logs record the execution status of the application's business logic; interface logs record HTTP / RPC calls between services; database logs record SQL execution status and database exceptions; and exception stack logs record the call stack information when the code throws an exception, which is the key to locating code-level defects.

[0102] In one possible embodiment, in S3, the standardized cleaning includes deduplicating duplicate logs, filtering noisy logs, completing missing key fields, standardizing timestamp formats, and mapping cross-node link IDs to the original logs.

[0103] Specifically, duplicate log deduplication is used to delete duplicate data caused by network retransmission, duplicate log output, etc., to avoid interfering with the analysis results; noisy log filtering is used to delete debugging logs, heartbeat logs and other information unrelated to defects, reducing the amount of data; missing key field completion is used to fix the problem of missing fields caused by incomplete log output; timestamp format standardization solves the problem of inconsistent timestamp formats in different systems; cross-node link ID association mapping is used to associate logs of different nodes on the same call link.

[0104] The formula for unifying the timestamp format is:

[0105]

[0106]

[0107] In the formula, This is the original timestamp. This is a timestamp conversion function. For standard Unix timestamps, For time formatting functions, To format the time string;

[0108] Specifically, different systems may output timestamps in different formats, such as local time, UTC time, or string format time, making direct time-series comparison impossible. This formula first converts all raw timestamps into a unified Unix timestamp for subsequent numerical calculations and time-series sorting; then, it formats the Unix timestamps into a human-readable string format for report display. Unix timestamps are cross-platform and easy to compute, making them the standard time format for log time-series analysis.

[0109] The cross-node link ID association mapping matching formula is as follows:

[0110]

[0111] In the formula, Logs for different nodes. To extract logs The unique identifier of the link, To associate the matching results, This indicates that the link association was successful. This indicates that the link association failed;

[0112] Specifically, in a distributed system, a user request passes through multiple service nodes, each generating its own independent logs. The traceId is a unique identifier generated at the request entry point and is passed to all downstream service nodes along with the request. This formula compares the traceIds of two log entries to determine if they belong to the same request's call chain, thus linking logs scattered across different nodes into a complete call chain.

[0113] The structured transformation process includes converting unstructured text logs into structured log data in JSON format, filtering out irrelevant logs, and retaining exception logs and main process logs.

[0114] Specifically, unstructured text logs are in plain string format, which machines cannot directly parse for their fields. JSON, on the other hand, is a lightweight data exchange format with a clear structure and is easily parsed by machines. Through structured transformation, fields such as timestamps, traceIds, and service names in the logs are extracted into JSON key-value pairs, facilitating subsequent feature matching and data analysis. Simultaneously, irrelevant logs are filtered out, retaining only exception logs and logs from key business processes, significantly reducing data processing volume and improving analysis efficiency.

[0115] In one possible embodiment, in S4, the abnormal feature library includes error codes, abnormal keywords, regular expressions, abnormal types, return code rules, timeout thresholds, and business verification rules.

[0116] Specifically, the anomaly feature library is a "knowledge base" for defect identification, storing the feature patterns of all known defects: error codes are predefined exception codes; exception keywords are common keywords in exception logs; regular expressions are used to match complex exception patterns; exception types are used to classify defects; return code rules define which business return codes belong to exceptions; timeout thresholds define the maximum allowed time for interface calls; and business validation rules define the legal scope of business logic.

[0117] The multi-dimensional matching includes system error code matching, abnormal keyword matching, regular expression matching, abnormal stack feature matching, business return code matching, and interface timeout threshold judgment.

[0118] Specifically, single-dimensional matching is prone to missed or false positives, while multi-dimensional matching, by combining multiple feature patterns, can significantly improve the accuracy of defect identification. For example, matching only the keyword "error" might misidentify the word "error" in normal logs, but by combining features such as error codes and exception stack traces, the true exception can be accurately identified.

[0119] The formula for determining the interface timeout threshold is as follows:

[0120] ,

[0121] In the formula, The time consumed by the call, For the requested time, For response time, This is the timeout threshold. Marked as timeout. The timeout was detected. Determined as normal;

[0122] Specifically, API timeout is a common system anomaly, manifested as a prolonged period without a response after a request is sent. This formula calculates the time difference between the API request and response to obtain the API call duration, and then compares it with a preset timeout threshold. If the duration exceeds the threshold, it is determined to be a timeout anomaly. The timeout threshold is typically set based on business requirements and system performance; different interfaces may have different thresholds.

[0123] The formula for matching logs with anomaly features similarity is:

[0124]

[0125]

[0126] In the formula, This is the content of a single structured log entry. For keyword / exception stack / regular expression matching templates in the exception feature library, This is the set of overlapping characters between the log and the feature template. This is a function for calculating character length. The similarity between logs and anomaly features. For similarity threshold, As an anomaly marker, For the hit defect, No abnormalities were found.

[0127] Specifically, for complex anomaly logs, such as anomaly stack traces, matching may fail due to subtle differences in log format. This formula uses character overlap to calculate the similarity between the log and the feature template. When the similarity reaches a preset threshold, it is determined to be an anomaly. This method has a certain degree of fault tolerance, can adapt to minor changes in log format, and improves the robustness of anomaly detection.

[0128] The formula for matching the service return code is:

[0129]

[0130] In the formula, For real-time return codes, This is a set of exception return codes. Mark the return code as abnormal;

[0131] Specifically, business systems typically use return codes to indicate the results of business execution, with non-successful return codes corresponding to different business exceptions. This formula quickly identifies business logic exceptions by determining whether the real-time return codes extracted from the logs belong to a predefined set of exception return codes. The set of exception return codes is usually predefined by business personnel according to business rules.

[0132] The defect types include null pointer exception, interface timeout exception, service internal error, business parameter exception, database operation exception, and business rule validation failure exception.

[0133] Specifically, the above defect types cover the most common exception scenarios in software systems: null pointer exceptions are common errors at the code level; interface timeout exceptions are caused by network or service performance issues; service internal errors are exceptions that are not handled by the server; business parameter exceptions are when the parameters passed by the client do not meet the requirements; database operation exceptions are caused by SQL execution or database connection problems; and business rule validation failures are when business logic execution fails.

[0134] In one possible embodiment, the anomaly feature library has a preset set of basic anomaly features covering general software systems, while also providing the function of user-defined business-specific anomaly features.

[0135] Specifically, the basic exception feature set includes common exception features for all software systems, such as Java null pointer exceptions and HTTP 500 errors, which can be used for defect identification in most systems. However, systems in different industries and with different business operations will have unique business exception features. Therefore, a customization function is provided, allowing users to add custom exception features according to their own business needs, thereby improving the applicability of the feature library.

[0136] The abnormal feature library also has a self-learning capability, extracting new abnormal features based on historical defect location results and updating the abnormal feature library regularly.

[0137] Specifically, the software system is constantly iterating and updating, and new defect types are constantly emerging. Relying solely on manual maintenance of the feature library cannot meet the needs. The self-learning capability analyzes historical defect location results, automatically extracts newly emerging abnormal features, and automatically adds them to the feature library after review, thereby achieving dynamic updates to the feature library and continuously improving the coverage of defect identification.

[0138] The frequency formula for the anomaly feature library is:

[0139]

[0140] In the formula, Add new abnormal features to the candidates. This is a historical defect log dataset. Statistical characteristics In the dataset The number of times it appears in Features Frequency of occurrence;

[0141] Specifically, not all newly appearing strings are valid anomaly features; only strings that frequently appear in historical defect logs are likely to be new anomaly features. This formula filters out high-frequency features as potential new anomaly features by statistically analyzing the frequency of candidate features in historical defect logs.

[0142] The formula for updating the anomaly feature database with newly added anomaly features is as follows:

[0143]

[0144] In the formula, The threshold for feature admission frequency. This is the original feature set of the anomaly feature library; To add new feature markers, To be included in the feature library and updated.

[0145] Specifically, this formula sets the admission criteria for new features: first, the frequency of the feature's occurrence must reach a preset threshold to ensure that the feature is a common anomalous feature; second, the feature must not be in the existing feature library to avoid duplicate addition. Only features that meet both of these conditions will be automatically added to the anomalous feature library.

[0146] In one possible embodiment, in S5, the specific process of defect localization is as follows: based on the traceId and spanId in the exception log, trace the complete distributed service call chain, combine the timestamp execution sequence of the log with the predefined module dependency path diagram of the system, locate the service name, module name, interface address, function method, code level / system configuration item where the defect occurred, and reverse deduce the exception propagation path and the scope of affected business modules.

[0147] Specifically, `traceId` identifies a complete distributed call, while `spanId` identifies a single node in the call chain. `traceId` allows you to find all logs for the same call, and `spanId` allows you to reconstruct the parent-child relationships of the calls, thus constructing a complete call chain diagram. Combining log timestamps determines the precise time and execution order of the exception, while combining this with the module dependency path diagram allows you to analyze the direction of exception propagation. Starting from the exception node and tracing the call chain backwards, you can find the initial trigger point (root cause) of the exception; traversing the call chain forwards identifies all downstream modules affected by the exception.

[0148] In one possible embodiment, the defect classification is also included. After the defect identification is completed, the defect score is calculated according to the comprehensive scoring formula. The identified defects are automatically classified into four levels: blocking, severe, general, and warning. The corresponding processing priority and response time limit are marked for each level.

[0149] Specifically, different defects have varying degrees of impact on the system, requiring different handling strategies. Defect grading quantifies the impact of defects, classifying them into different levels. This allows testing and development teams to prioritize critical defects, improving problem-solving efficiency. Processing priorities and response timelines clearly define the handling requirements for different defect levels, ensuring that critical defects are fixed promptly.

[0150] The comprehensive scoring formula is as follows:

[0151]

[0152] In the formula, Score based on the scope of business impact. Assign a score to the system crash risk. The degree of business disruption is scored. Score the user's perceived priority. The weights are fixed.

[0153] Specifically, this formula employs a multi-dimensional weighted scoring method to comprehensively assess the severity of defects from four main dimensions: Business Impact Scope reflects the number of business modules affected by the defect; System Crash Risk reflects the likelihood of the defect causing a complete system crash; Business Interruption Degree reflects the extent to which the defect disrupts major business processes; and User Perception Priority reflects the degree of impact of the defect on user experience. The weights of the four dimensions are set according to business needs, with the main business dimensions typically having higher weights.

[0154] In one possible embodiment, in S6, the standardized defect location report includes test case name, test execution time, test case execution result, defect unique identifier, defect level, defect type, location, related abnormal log fragments, abnormal propagation chain diagram, and suggested handling direction;

[0155] Specifically, each field in the report has a clear function: the test case name and execution time are used to locate the test scenario in which the defect occurred; the defect unique identifier is used to globally and uniquely identify the defect; the defect level and type are used to quickly understand the severity and nature of the defect; the location is used by developers to quickly find the defective code; the relevant log fragments and propagation chain diagram are used to reconstruct the complete context of the defect occurrence; and the suggested handling direction provides developers with a reference for remediation.

[0156] The formula for encoding the unique identifier of the defect is:

[0157]

[0158] In the formula, As a globally unique identifier for defects, Each test case has a unique identifier. To standardize Unix timestamps, For hash functions;

[0159] Specifically, this formula uses a three-segment encoding to generate a globally unique defect ID. The test case number associates the defect with the test case that triggered it; the timestamp ensures that defects triggered by the same test case at different times have different IDs; and the MD5 hash function converts the traceId into a fixed-length string, further guaranteeing the uniqueness of the ID. This encoding method not only guarantees the global uniqueness of the ID but also includes the defect's traceability information.

[0160] Once the standardized defect location report is generated, it is pushed to the defect management system and associated with the execution records and test screenshots of the corresponding test cases.

[0161] Specifically, reports are automatically pushed to the defect management system, enabling end-to-end logical management of defects from discovery to repair. Linking test case execution records and test screenshots facilitates rapid defect reproduction by developers, improving repair efficiency. Simultaneously, all defect information is centrally stored in the defect management system, facilitating subsequent statistical analysis and quality traceability.

[0162] The application scenarios of this invention are as follows:

[0163] Microservice interface automated testing scenario: Applicable to the interface testing phase of microservice architecture systems such as enterprise business middleware, distributed business systems, and cloud service platforms. Adapted to system operation logs, interface request and response logs, microservice link logs, and error stack log formats. When the test client executes batch automated interface test cases, the system synchronously collects all multi-source logs in real time and performs standardized cleaning and structure transformation such as deduplication, noise reduction, and field completion. Relying on the exception feature library, it performs multi-dimensional matching through error codes, keywords, and regular expressions to automatically identify defects such as timeouts, parameter anomalies, and service errors. Combined with traceId link tracing, execution sequence, and module dependency relationships, it locates the service, interface, and method levels and completes automatic defect classification. Finally, it generates a standardized defect report associated with the test cases, realizing unattended operation of the entire interface testing process and solving the problems of scattered microservice node logs, difficulty in tracing the root cause of interface failures, and low efficiency of manual log investigation.

[0164] End-to-end automated testing scenarios for web systems: Applicable to end-to-end testing of front-end and back-end linked web applications such as e-commerce platforms, collaborative office systems, and government portals. Adapts to test execution logs, application running logs, interface interaction logs, and exception stack traces. The system runs UI automated test cases and collects full-link logs simultaneously. After unstructured logs are formatted and associated with link IDs, they are converted into JSON structured data. Through an exception feature library, it identifies issues such as page interaction anomalies, business process verification failures, and service call anomalies. Based on the distributed call chain, it reverse-engineers the exception propagation path, accurately locates the module and code level to which the defect belongs, automatically classifies the defect level, and outputs a complete test report. It meets the full-function automated verification requirements of web systems and reduces the reliance on manual defect location in end-to-end testing, as well as the probability of misjudgment and missed judgment.

[0165] Intelligent automation equipment factory testing scenarios: Applicable to the factory quality inspection of integrated hardware and software intelligent equipment such as industrial control equipment, intelligent self-service terminals, intelligent warehousing equipment, and building automation equipment. Adapted to equipment operation logs, hardware status logs, communication interface logs, and equipment crash stack log formats; The test platform batch executes automated test cases for equipment functions, synchronously collects raw logs from all dimensions of the equipment and performs standardized cleaning, uses an anomaly feature library to identify defects such as equipment crashes, response timeouts, function execution anomalies, and communication failures, and combines module dependency path graphs to locate fault locations such as embedded programs, hardware drivers, and configuration items, automatically generates defect location reports and binds corresponding test items, realizing full automation of the intelligent equipment factory testing process, effectively improving the equipment defect detection rate and shortening the overall testing cycle.

[0166] Specialized testing scenarios for distributed database systems: Applicable to specialized performance and functional testing of data storage systems such as big data platforms, distributed data warehouses, and business database clusters. It adapts to database operation logs, cluster node operation logs, SQL execution logs, and database exception stack traces. When executing automated test cases for database read / write, concurrent transactions, and data synchronization, the system collects logs from each cluster node in real time and completes cross-node link ID association mapping. Based on timeout thresholds, return code rules, and business verification rules, it identifies defects such as slow queries, connection interruptions, and abnormal data operations. Relying on log time-series analysis and cluster dependency relationships, it locates faulty nodes and abnormal SQL statements, automatically completes defect classification, and outputs test reports. This addresses the pain points of numerous distributed database nodes, fragmented logs, and the difficulty of manually locating data-related faults, ensuring the stability of the database cluster operation.

[0167] Industrial IoT terminal batch regression testing scenario: Applicable to the regression testing of batch terminal devices such as factory sensing terminals, remote monitoring equipment, and outdoor IoT gateways after version iteration. Adapted to terminal operation logs, network communication logs, data reporting logs, and abnormal error log formats; the operation and maintenance test terminal batches and distributes regression test cases. The system collects multi-source logs from massive terminal devices in parallel and performs unified cleaning and structuring processing. Relying on an anomaly feature library with feature matching capabilities, it batch screens common and individual defects such as communication interruption, data reporting anomalies, and device offline. It traces the source of anomalies through complete call links, automatically distinguishes four defect levels: blocking, severe, general, and warning, and summarizes batch test reports. There is no need for manual log review of each device, which greatly improves the efficiency of batch regression testing of IoT devices and avoids the risk of hidden faults introduced by version iteration.

[0168] It should be noted that the method of this embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this embodiment, and the multiple devices will interact with each other to complete the described automated device defect location testing method based on log analysis.

[0169] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0170] Example 2

[0171] See Figure 3 Embodiment 2 of the present invention also provides an automated defect location testing device based on log analysis, which employs the above-described automated defect location testing method based on log analysis, including:

[0172] Configuration module 100: Used to configure log collection rules and establish a unified multi-source log access standard;

[0173] Log collection module 200: Used to execute automated test cases and collect raw multi-source logs in real time;

[0174] Log cleaning module 300: used to perform standardized cleaning and structured transformation processing on the original multi-source logs to generate structured log data;

[0175] Defect identification module 400: used to perform multi-dimensional matching on the structured log data based on the abnormal feature library to identify defect types and abnormal events;

[0176] Defect localization module 500: Used to locate the location, root cause category, and scope of impact of defects based on service call chain, log execution sequence relationship, and system module dependency path;

[0177] Report generation module 600: Used to integrate defect identification and location results, generate a standardized defect location report, and associate the standardized defect location report with the execution results of the corresponding test cases.

[0178] It should be noted that the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0179] Example 3

[0180] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code of an automated defect location testing method based on log analysis. The program code includes instructions for executing the automated defect location testing method based on log analysis of Embodiment 1 or any possible implementation thereof.

[0181] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0182] Example 4

[0183] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0184] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the log analysis-based automated defect location testing method of Embodiment 1 or any possible implementation thereof by calling the program instructions.

[0185] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0186] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0187] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0188] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. An automated defect location testing method based on log analysis, characterized in that, include: S1. Configure log collection rules and establish a unified multi-source log access standard; S2. Execute automated test cases and collect raw multi-source logs in real time; S3. Perform standardized cleaning and structured transformation on the original multi-source logs to generate structured log data; S4. Perform multi-dimensional matching on the structured log data based on the anomaly feature library to identify defect types and abnormal events; S5. Based on the service call chain, log execution sequence relationship, and system module dependency path, locate the location of the defect, the root cause category, and the scope of impact; S6. Integrate the defect identification and location results, generate a standardized defect location report, and associate the standardized defect location report with the execution results of the corresponding test cases.

2. The automated defect location testing method based on log analysis according to claim 1, characterized in that, In S1, the log collection rules include the log path, log type, collection fields, and collection frequency of the system under test; The log types include runtime logs, API logs, and error logs; The collected fields include timestamp, traceId, service name, level, content, IP, and port; The multi-source logs include system operation logs, interface request and response logs, database logs, microservice link logs, and error stack logs.

3. The automated defect location testing method based on log analysis according to claim 1, characterized in that, In S2, the original multi-source logs include test logs, application logs, interface logs, database logs, and exception stack logs.

4. The automated defect location testing method based on log analysis according to claim 1, characterized in that, In S3, the standardized cleaning includes deduplication of duplicate logs, filtering of noisy logs, completion of missing key fields, standardization of timestamp format, and cross-node link ID association mapping of the original logs. The formula for unifying the timestamp format is: In the formula, This is the original timestamp. This is a timestamp conversion function. For standard Unix timestamps, For time formatting functions, To format the time string; The cross-node link ID association mapping matching formula is as follows: In the formula, Logs for different nodes. To extract logs The unique identifier of the link, To associate the matching results, This indicates that the link association was successful. This indicates that the link association failed; The structured transformation process includes converting unstructured text logs into structured log data in JSON format, filtering out irrelevant logs, and retaining exception logs and main process logs.

5. The automated defect location testing method based on log analysis according to claim 1, characterized in that, In S4, the abnormal feature library includes error codes, abnormal keywords, regular expressions, abnormal types, return code rules, timeout thresholds, and business verification rules. The multi-dimensional matching includes system error code matching, abnormal keyword matching, regular expression matching, abnormal stack feature matching, business return code matching, and interface timeout threshold judgment. The formula for determining the interface timeout threshold is as follows: 、 In the formula, The time consumed by the call, For the requested time, For response time, This is the timeout threshold. Marked as timeout. The timeout was detected. Determined as normal; The formula for matching logs with anomaly features similarity is: In the formula, This is the content of a single structured log entry. For keyword / exception stack / regular expression matching templates in the exception feature library, This is the set of overlapping characters between the log and the feature template. This is a function for calculating character length. The similarity between logs and anomaly features. For similarity threshold, As an anomaly marker, For the hit defect, No abnormalities were found. The formula for matching the service return code is: In the formula, For real-time return codes, This is a set of exception return codes. Mark the return code as abnormal; The defect types include null pointer exception, interface timeout exception, service internal error, business parameter exception, database operation exception, and business rule validation failure exception.

6. The automated defect location testing method based on log analysis according to claim 5, characterized in that, The anomaly feature library has a pre-set set of basic anomaly features covering general software systems, and also provides the function of user-defined business-specific anomaly features. The abnormal feature library also has a self-learning capability, extracting new abnormal features based on historical defect location results and updating the abnormal feature library regularly. The frequency formula for the anomaly feature library is: In the formula, Add new abnormal features to the candidates. This is a historical defect log dataset. Statistical characteristics In the dataset The number of times it appears in Features Frequency of occurrence; The formula for updating the anomaly feature database with newly added anomaly features is as follows: In the formula, The threshold for feature admission frequency. This is the original feature set of the anomaly feature library; To add new feature markers, To be included in the feature library and updated.

7. The automated defect location testing method based on log analysis according to claim 1, characterized in that, In S5, the specific process of defect localization is as follows: based on the traceId and spanId in the exception log, trace the complete distributed service call chain, combine the timestamp execution sequence of the log with the predefined module dependency path diagram of the system, locate the service name, module name, interface address, function method, code level / system configuration item where the defect occurred, and reverse deduce the exception propagation path and the scope of affected business modules.

8. The automated defect location testing method based on log analysis according to claim 7, characterized in that, It also includes defect classification. After defect identification is completed, the defect score is calculated according to the comprehensive scoring formula. The identified defects are automatically divided into four levels: blocking, severe, general, and warning. Each level is marked with the corresponding processing priority and response time limit. The comprehensive scoring formula is as follows: In the formula, Score based on the scope of business impact. Assign a score to the system crash risk. The degree of business disruption is scored. Score the user's perceived priority. The weights are fixed.

9. The automated defect location testing method based on log analysis according to claim 1, characterized in that, In S6, the standardized defect location report includes test case name, test execution time, test case execution result, defect unique identifier, defect level, defect type, location, related abnormal log fragments, abnormal propagation chain diagram, and suggested handling direction; The formula for encoding the unique identifier of the defect is: In the formula, As a globally unique identifier for defects, Each test case has a unique identifier. To standardize Unix timestamps, For hash functions; Once the standardized defect location report is generated, it is pushed to the defect management system and associated with the execution records and test screenshots of the corresponding test cases.

10. An automated defect location testing device based on log analysis, employing the automated defect location testing method based on log analysis as described in any one of claims 1-9, characterized in that, include: Configuration module: Used to configure log collection rules and establish a unified multi-source log access standard; Log collection module: Used to execute automated test cases and collect raw multi-source logs in real time; Log cleaning module: used to perform standardized cleaning and structured transformation on the original multi-source logs to generate structured log data; Defect identification module: used to perform multi-dimensional matching on the structured log data based on the anomaly feature library to identify defect types and abnormal events; Defect localization module: used to locate the location, root cause category, and scope of impact of defects based on service call chain, log execution sequence relationship, and system module dependency path; Report generation module: Used to integrate defect identification and location results, generate standardized defect location reports, and associate the standardized defect location reports with the execution results of corresponding test cases.

Images