An Automated Testing Hierarchical Feedback and Intelligent Attribution Method and System Based on Multi-Source Data Fusion

Abstract

This invention discloses an automated testing hierarchical feedback and intelligent attribution method and system based on multi-source data fusion, belonging to the field of automated testing technology. This invention collects multi-source data from the build domain, test domain, and environment domain, forming a test evidence chain through globally unique identifiers; it automatically and intelligently attributes failure logs to causes based on a fault feature rule base; it employs a hierarchical feedback mechanism of L1 real-time blocking feedback and L2 periodic deep reporting; and it automatically recommends suspects when attribution is to code defects. This invention can significantly shorten the problem localization cycle, achieve accurate early warning and efficient attribution, improve the efficiency of automated testing and defect repair, and is suitable for automated testing quality control in continuous integration scenarios.

Description

Technical Field

[0001] This invention relates to the technical field of automated testing, and more particularly to an automated testing hierarchical feedback and intelligent attribution method and system based on multi-source data fusion. Background Technology

[0002] In continuous integration automated testing processes, generating test reports after test execution is an industry standard practice. Existing traditional test report systems based on tools such as Allure and JUnit generally suffer from the following technical limitations: results are simply listed, with reports primarily displaying test case-level pass or fail statuses along with error stack traces; information dimensions are limited, containing only test execution logs and lacking contextual data such as code commits, build environment, and hardware status; failure attribution relies entirely on manual log reading and experience-based judgment, resulting in low analysis efficiency; and feedback methods are rudimentary, with reports sent out in a uniform, mass manner without targeted, tiered alerts.

[0003] The aforementioned traditional solutions lead to information overload for R&D personnel, difficulties in troubleshooting, inability to automatically classify the root causes of failures, untimely notification of serious problems, and a tendency for alarm fatigue. Furthermore, test results are isolated from code changes and environmental status data, making it difficult to form a complete chain of evidence, significantly extending the defect repair cycle, and reducing the efficiency of R&D and testing collaboration. Summary of the Invention

[0004] This invention addresses the shortcomings of existing test reports, which simply list the pass / fail status of test cases and provide stack trace information. These issues include limited information dimensions, reliance on manual attribution, inefficient feedback, and low troubleshooting efficiency. The technical solution adopted by this invention is as follows: In a first aspect of the present invention, an automated test hierarchical feedback and intelligent attribution method based on multi-source data fusion is provided, comprising the following steps: S1. Collect construction domain data, test domain data, and environment domain data throughout the automated testing process, and link and merge the three types of data through a globally unique identifier to obtain the test evidence chain; S2. Load the predefined fault feature rule library, perform regular expression or keyword matching on the test failure logs, and classify the failure results; S3. For blocking-level failures, perform L1 real-time blocking feedback processing; for overall quality-level failures, generate L2 periodic in-depth analysis reports. S4. When a failure is attributed to a code defect, retrieve the code change records of the version control system, calculate the suspicion index, and recommend the person responsible for investigation.

[0005] Preferably, the build domain data includes Commit ID, code commit author, build start and end times, and compilation status; The test domain data includes test case execution logs, Allure report attachments, performance data, and execution screenshots; The environmental domain data includes Agent node, CPU / memory load, network latency, and the status of the device under test.

[0006] Preferably, the fault feature rule base is a dynamic rule base that is updated online, and the matching rules include: The "Connection refused" and "Timeout" errors are attributed to environmental / network issues; the "AssertionError" and "NullPointerException" errors are attributed to code defects; and the "Equipment not found" and "Flash failed" errors are attributed to hardware / device issues.

[0007] Preferably, the L1 real-time blocking feedback includes real-time monitoring logs during test execution, identifying when a blocking-level test case fails, stopping subsequent non-critical tests, and sending an immediate alert to the code submission author; The L2 periodic in-depth analysis report includes a comprehensive report generated at a preset cycle, containing trend charts, Top N failed use cases, defect distribution, and resource health status, which is then pushed to project and quality management personnel.

[0008] More preferably, the triggering conditions for the L1 real-time feedback include a surge in the failure rate and consecutive failures of specific modules.

[0009] As a preferred approach, code commit records within a preset time window are retrieved based on the module / file path of the failed test case; a suspicion index is calculated based on the relevance of the modified file and the time of modification; and the recommended investigator, suspicion index, and commit record link are displayed in the test report.

[0010] In another aspect of the present invention, an automated test hierarchical feedback and intelligent attribution system based on multi-source data fusion is also provided, applied to the aforementioned automated test hierarchical feedback and intelligent attribution method based on multi-source data fusion, comprising: The multi-source data fusion module is used to collect data from the build domain, test domain, and environment domain and fuse them together using a unique identifier; The intelligent attribution module is used to automatically classify failure logs based on a fault feature rule base. The hierarchical feedback module is used to perform L1 real-time blocking feedback and generate and push L2 periodic reports; The responsible person recommendation module is used to calculate and output the responsible person for investigation based on the code change record.

[0011] Preferably, the intelligent attribution module supports online updates of the fault feature rule base, automatically classifying failure results into one of the following: environmental / network problems, code defects, and hardware / equipment problems.

[0012] Preferably, the hierarchical feedback module connects to an instant messaging tool to send real-time alarms for blocking-level failures.

[0013] Preferably, the responsible person recommendation module supports code call chain analysis to improve recommendation accuracy.

[0014] Compared with the prior art, the present invention has the following significant advantages: Improve troubleshooting efficiency: By using intelligent attribution, the raw stack logs are transformed into clear categories such as environment / network, code defects, and hardware devices, overcoming the pain point of manually reviewing logs one by one and quickly locating the problem direction.

[0015] Achieving precise and efficient feedback: The hierarchical feedback mechanism ensures that obstructive issues reach the responsible parties in real time, while routine issues are aggregated and presented, avoiding irrelevant alarms and alarm fatigue, and solving the shortcomings of traditional reporting and feedback that are rough and untimely.

[0016] Build a complete and traceable chain of evidence: Multi-source data fusion and unique identifier association break down data silos and globally connect test results with code submissions, build environments, and device status to support in-depth root cause analysis.

[0017] Accelerate defect closure and collaboration: Dynamic responsibility recommendations automatically associate code modifiers, clarify the person responsible for investigation, shorten the problem follow-up and repair process, and improve team collaboration and R&D delivery efficiency.

[0018] Supporting macro-level quality control: Periodic in-depth analysis reports provide data such as pass rate trends, defect distribution, and resource health, providing an objective basis for project quality assessment and decision-making. Attached Figure Description

[0019] Figure 1 This is a flowchart of an automated testing hierarchical feedback and intelligent attribution method based on multi-source data fusion in a specific embodiment of the present invention; Figure 2 This is a framework diagram of an automated testing hierarchical feedback and intelligent attribution system based on multi-source data fusion in a specific embodiment of the present invention; Detailed Implementation The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] refer to Figure 1 As shown, this embodiment provides an automated test hierarchical feedback and intelligent attribution method based on multi-source data fusion. The specific steps are as follows, and a detailed explanation is provided in conjunction with the home fitness scenario to adapt to users of different ages and body types: S1: Collect construction domain data, test domain data, and environment domain data throughout the automated testing process. Use a globally unique identifier to link and fuse these three types of data to obtain a test evidence chain, specifically including: Three types of data were collected via the interface: Build domain data: Git Commit ID, committer, build start / end time, compilation results; Test domain data includes: test case execution logs, error stack traces, Allure report attachments, execution screenshots, and performance metrics. Environmental domain data: This includes CPU utilization, memory load, network latency, and device connectivity status of the test agent nodes. The system uses the build_id of this task as a globally unique identifier to bind and store the above three types of data, forming a complete and traceable chain of test evidence.

[0021] S2: Load a predefined fault feature rule base, perform regular expression or keyword matching on the test failure logs, and categorize the failure results, specifically including: Load a predefined fault characteristic rule base and perform regular expression matching on the logs of failed test cases: If the log contains "Connection refused", "Timeout", or "Network error", it will be automatically marked as an environment / network problem. If the log contains AssertionError, NullPointerException, or IllegalArgumentException, mark it as a code defect; If the logs contain "Device not found," "Flash failed," or "Offline," mark it as a hardware / device issue. The attribution results are highlighted at the top of the report, eliminating the need for manual line-by-line log analysis.

[0022] S3: For blocking-level failures, perform L1 real-time blocking feedback processing; for overall quality-level failures, generate L2 periodic in-depth analysis reports, specifically including: Execution is achieved through a hierarchical feedback closed-loop mechanism: L1 Real-time Interruption Feedback: During test execution, the monitoring module scans the logs in real time. When a core smoke test case fails, the execution of subsequent non-critical test cases is immediately interrupted; an alert is pushed to the code committer via DingTalk / WeChat Work robot, including: test case name, failure attribution, log summary, and report link.

[0023] L2 Periodic In-Depth Analysis Report: A statistical report is automatically generated every morning, including: build success rate trend, test pass rate changes, top 10 frequently failed test cases, defect distribution by module, and test environment resource health. The report is sent to the project manager, test manager, and quality engineer for quality review and decision-making.

[0024] S4: When a failure is attributed to a code defect, retrieve the code change records from the version control system, calculate the suspicion index, and recommend the responsible party for investigation, specifically including: Based on the code module / file path corresponding to the failed test case, retrieve Git commit records within the last 24 hours; calculate the suspicion index based on the relevance of the modified files and the proximity of the commit time; recommend 1-2 priority investigators in the test report, and attach the commit ID and code change link to directly locate suspicious modifications.

[0025] The formula for calculating the suspicion index is as follows: S = α·Rfile+β·Dtime+γ·Hdefect; Where: Rfile represents the file modification relevance: it is the Jaccard similarity coefficient between the set of files modified by the submitter and the set of files associated with the failed test cases, i.e., Rfile=|Fcommit∩Ffail| / |Fcommit∪Ffail|, with a value range of [0,1]. The higher the overlap between the files modified by the submitter and the files involved in the failed test cases, the higher the score of this item. Dtime represents the submission time decay factor: it uses the exponential decay function Dtime=e The calculation is λ·Δt, where Δt is the time difference (in hours) between the submission time and the time the test failed, and λ is the decay coefficient (default value is 0.1). The closer the submission is to the time of failure, the higher the score for this item. Hdefect represents the historical defect rate: it is the percentage of times that the code submitted by the submitter caused test failures in the most recent N iterations (default N=5), that is, Hdefect = number of failures caused by the submitter / total number of submissions by the submitter, and the value range is [0,1]. α, β, and γ are weighting coefficients that satisfy α+β+γ=1. The default values ​​are α=0.5, β=0.3, and γ=0.2. The system supports parameter optimization based on the actual situation of the team.

[0026] Please see Figure 2 As shown, in a second aspect of the present invention, an automated test hierarchical feedback and intelligent attribution system based on multi-source data fusion is proposed, applied to the aforementioned automated test hierarchical feedback and intelligent attribution method based on multi-source data fusion, comprising: The multi-source data fusion module is used to access the build platform, test execution engine, and infrastructure monitoring, collect data from the build domain, test domain, and environment domain, and associate them uniformly with task_id / build_id to form a test evidence chain.

[0027] The intelligent attribution module has a built-in fault feature rule base that supports keyword / regular expression matching and automatically classifies failure logs into environmental issues, code defects, and hardware problems; the rule base supports online learning and dynamic updates.

[0028] The hierarchical feedback module includes a real-time monitoring unit: used to identify blocking-level failures during testing, trigger process interruptions and immediate alarms; Periodic Reporting Unit: Used to generate quality analysis reports according to a preset cycle and push them to management.

[0029] The responsible person recommendation module is used to connect with version control systems such as Git. It calculates the suspicion index based on code change records, outputs the recommended responsible persons for investigation, and supports combining call chain analysis to improve accuracy.

[0030] The report display and storage module is used to generate visual test reports, displaying attribution results, evidence chains, investigator recommendations, trend charts, and supporting historical data query and export.

[0031] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An automated testing hierarchical feedback and intelligent attribution method based on multi-source data fusion, characterized in that, Includes the following steps: S1. Collect construction domain data, test domain data, and environment domain data throughout the automated testing process, and link and merge the three types of data through a globally unique identifier to obtain the test evidence chain; S2. Load the predefined fault feature rule library, perform regular expression or keyword matching on the test failure logs, and classify the failure results; S3. For blocking-level failures, perform L1 real-time blocking feedback processing; for overall quality-level failures, generate L2 periodic in-depth analysis reports. S4. When a failure is attributed to a code defect, retrieve the code change records of the version control system, calculate the suspicion index, and recommend the person responsible for investigation.

2. The automated testing hierarchical feedback and intelligent attribution method based on multi-source data fusion according to claim 1, characterized in that, The build domain data includes the Commit ID, code commit author, build start and end times, and compilation status. The test domain data includes test case execution logs, Allure report attachments, performance data, and execution screenshots; The environmental domain data includes Agent node, CPU / memory load, network latency, and the status of the device under test.

3. The automated testing hierarchical feedback and intelligent attribution method based on multi-source data fusion according to claim 1, characterized in that, The fault feature rule base is a dynamic rule base that is updated online, and the matching rules include: The "Connection refused" and "Timeout" errors are attributed to environmental / network issues; the "AssertionError" and "NullPointerException" errors are attributed to code defects; and the "Equipment not found" and "Flash failed" errors are attributed to hardware / device issues.

4. The automated testing hierarchical feedback and intelligent attribution method based on multi-source data fusion according to claim 1, characterized in that, The L1 real-time blocking feedback includes real-time monitoring logs during test execution, identifying when a blocking-level test case fails, stopping subsequent non-critical tests, and sending an immediate alert to the code submission author. The L2 periodic in-depth analysis report includes a comprehensive report generated at a preset cycle, containing trend charts, Top N failed use cases, defect distribution, and resource health status, which is then pushed to project and quality management personnel.

5. The automated testing hierarchical feedback and intelligent attribution method based on multi-source data fusion according to claim 4, characterized in that, The triggering conditions for the L1 real-time feedback include a surge in the failure rate and consecutive failures of specific modules.

6. The automated testing hierarchical feedback and intelligent attribution method based on multi-source data fusion according to claim 1, characterized in that, Based on the module / file path of the failed test case, retrieve code commit records within a preset time window; calculate the suspicion index based on the relevance of the modified file and the time of modification; display recommended investigators, suspicion index, and commit record links in the test report.

7. An automated testing hierarchical feedback and intelligent attribution system based on multi-source data fusion, used to implement the method of any one of claims 1-6, characterized in that, include: The multi-source data fusion module is used to collect data from the build domain, test domain, and environment domain and fuse them together using a unique identifier; The intelligent attribution module is used to automatically classify failure logs based on a fault feature rule base. The hierarchical feedback module is used to perform L1 real-time blocking feedback and generate and push L2 periodic reports; The responsible person recommendation module is used to calculate and output the responsible person for investigation based on the code change record.

8. The automated testing hierarchical feedback and intelligent attribution system based on multi-source data fusion according to claim 7, characterized in that, The intelligent attribution module supports online updates of the fault feature rule base, automatically classifying failure results into one of the following: environmental / network problems, code defects, and hardware / equipment problems.

9. The automated testing hierarchical feedback and intelligent attribution system based on multi-source data fusion according to claim 7, characterized in that, The hierarchical feedback module connects to instant messaging tools to send real-time alarms for blocking-level failures.

10. The automated testing hierarchical feedback and intelligent attribution system based on multi-source data fusion according to claim 7, characterized in that, The responsible person recommendation module supports code call chain analysis to improve recommendation accuracy.

Images