Testing methods, devices and electronic equipment for attributing anomalies

Abstract

This disclosure provides a method, apparatus, and electronic device for attributing test anomalies, relating to the field of computer technology, particularly artificial intelligence, large-scale models, natural language processing, intelligent agents, deep learning, and mapping. The specific implementation scheme is as follows: Failed test cases corresponding to the failed target test task are clustered to obtain at least one cluster; for any cluster, multi-source operation and maintenance data related to the failed test cases within the cluster are acquired; using a large-scale model, based on the multi-source operation and maintenance data, failure attribution analysis is performed on the failed test cases within the cluster to obtain the failure cause analysis results of the target test task; wherein, the failure cause analysis results include the test failure root cause type and attribution description information.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, large models, natural language processing, intelligent agents, deep learning, and mapping, and especially to a method, apparatus, and electronic device for testing anomaly attribution. Background Technology

[0002] In some application scenarios, such as CI (Continuous Integration) / CD (Continuous Delivery / Deployment) processes, automated testing is a key factor in ensuring software version quality and improving iteration efficiency. When performing large-scale and high-frequency automated testing tasks, test failure attribution becomes particularly crucial. Test failure attribution can pinpoint the root cause of problems, providing relevant personnel with clear and specific fault clues, helping them quickly identify the fault and avoid getting bogged down in blind troubleshooting. In this way, relevant personnel can focus their efforts on the actual problem, significantly shortening the problem-solving time and ultimately effectively improving the efficiency of the entire R&D delivery process.

[0003] Therefore, it is important to know how to attribute test failures. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and electronic device for testing anomaly attribution.

[0005] According to one aspect of this disclosure, a test anomaly attribution method is provided. The method includes: clustering failed test cases corresponding to a target test task that has failed to pass, to obtain at least one cluster; for any cluster, acquiring multi-source operation and maintenance data related to the failed test cases in the cluster; using a large model, performing failure attribution analysis on the failed test cases in the cluster based on the multi-source operation and maintenance data, to obtain the failure cause analysis result of the target test task; wherein, the failure cause analysis result includes test failure root cause type and attribution description information.

[0006] According to another aspect of this disclosure, a test anomaly attribution apparatus is provided. The apparatus includes: a clustering module for clustering failed test cases corresponding to a target test task that has failed, to obtain at least one cluster; a first acquisition module for acquiring multi-source operation and maintenance data related to the failed test cases in any of the clusters; and a first analysis module for using a large model to perform failure attribution analysis on the failed test cases in the clusters based on the multi-source operation and maintenance data, to obtain a failure cause analysis result for the target test task; wherein the failure cause analysis result includes test failure attribution type and attribution description information.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the test anomaly attribution method proposed in the foregoing aspect of this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to execute the test anomaly attribution method proposed in the foregoing aspect of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the test anomaly attribution method proposed in the foregoing aspect of this disclosure.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure; Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure; Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of the LLM-based intelligent attribution agent for continuous integration test case failures provided in this disclosure; Figure 5 This is a schematic diagram according to the fourth embodiment of the present disclosure; Figure 6 This is a block diagram of an electronic device used to implement the test anomaly attribution method of the embodiments of this disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] Although automated testing is widely used in continuous integration / continuous delivery processes, the analysis of test failures and problem localization still face several challenges, which seriously affect R&D delivery efficiency and problem-solving progress.

[0015] First, the efficiency of problem localization is low. In the CI / CD process, after the execution of automated test cases fails, developers and testers need to spend a lot of effort on manual analysis, resulting in long problem localization cycles and delayed feedback.

[0016] Secondly, the data silo effect is prominent, lacking a unified system that can efficiently link and integrate heterogeneous data from different sources (including code repositories, CI / CD platforms, monitoring platforms, service environments, service databases, etc.), making it difficult for relevant information to be linked efficiently and increasing the difficulty of troubleshooting.

[0017] At the same time, the failure analysis and problem-solving process is highly dependent on experience and relies heavily on senior personnel with extensive business and technical experience. This sets a high barrier to entry for newly hired team members, which is not conducive to improving the overall efficiency of the team's troubleshooting.

[0018] In addition, manually troubleshooting occasional failures and complex environment configuration issues consumes a lot of testing and R&D resources, resulting in high human resource costs and further restricting the improvement of R&D efficiency.

[0019] In addition to the problems mentioned above, the troubleshooting methods in related technologies also have obvious shortcomings. Troubleshooting methods based on fixed scripts or regular expressions cannot understand log content in natural language format, nor can they interpret the semantic impact and potential functional overlap brought about by code changes, making it difficult to deal with complex failure scenarios.

[0020] Furthermore, non-LLM machine learning models in related technologies also have limitations. Although these models can perform simple fault classification, they are limited in their ability to handle complex unstructured data and lack interpretability of the attribution process and the ability to extract evidence chains, thus failing to provide clear and traceable evidence for problem solving.

[0021] To address at least one of the aforementioned problems, this disclosure proposes a method, apparatus, and electronic device for testing anomaly attribution.

[0022] Figure 1 The diagram is based on the first embodiment of this disclosure. It should be noted that the test anomaly attribution method of this disclosure can be applied to a test anomaly attribution device, which can be configured in an electronic device so that the electronic device can perform test anomaly attribution function.

[0023] Among them, electronic devices can be any device with computing capabilities, such as personal computers (PCs), mobile terminals, servers, etc. Mobile terminals can be, for example, in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, smart speakers, servers, server clusters, and other hardware devices with various operating systems, touch screens and / or displays.

[0024] The test anomaly attribution device can also be software within an electronic device, such as test anomaly attribution software. The following embodiments use an electronic device as an example for illustration.

[0025] like Figure 1 As shown, this test anomaly attribution method may include the following steps: Step 101: Cluster the failed test cases corresponding to the failed target test tasks to obtain at least one cluster.

[0026] Optionally, the target test task may include, but is not limited to, CI (Continuous Integration) / CD (Continuous Delivery / Deployment) interface automation testing, performance testing, regression testing, pre-release testing, etc. It should be noted that the target test task can be a task that fails or encounters an exception, and the target test task may include test cases that fail, referred to as failed test cases in this disclosure.

[0027] It should also be noted that the failed test case corresponding to the target test task that failed can be, but is not limited to, one, and this disclosure does not impose any restrictions on this.

[0028] In this embodiment of the disclosure, failed test cases corresponding to the failed target test tasks can be clustered to obtain at least one cluster.

[0029] As an example, failed test cases can be clustered according to error messages. For instance, assuming the error message is a stack trace, failed test cases with high similarity in stack traces can be identified as belonging to the same cluster. Or, assuming the error message includes an error type, the error message can be parsed to obtain the error type, and failed test cases with the same error type can be identified as belonging to the same cluster.

[0030] As another example, failed test cases can be clustered according to interface information; for example, failed test cases with the same interface name during execution can be identified as failed test cases belonging to the same cluster; the relationship between failed test cases and corresponding interfaces includes, but is not limited to: the execution of failed test cases can depend on the corresponding interface, failed test cases are used to test the corresponding interface, and failed test cases call the corresponding interface during execution.

[0031] As another example, failed test cases can be clustered according to environmental dimensions (such as the container used, time zone, etc.). For instance, failed test cases executed in the same container can be identified as belonging to the same cluster. It should be noted that when there are a large number of failed test cases in a certain container, it is likely that there is a problem with the container itself. For example, the version of a critical software installed in the container may be too low, causing some functions to not work properly, thus causing test cases that depend on these functions to fail. Or, the container resources may be insufficient, such as excessive memory or CPU (Central Processing Unit) usage, causing test cases to malfunction due to resource contention during execution. Another example is that failed test cases executed in test environments within the same time zone can be grouped together. It should be noted that if a large number of time-related test cases fail in a test environment in a certain time zone, it may be that the time processing logic of the execution entity in this disclosure is flawed in that time zone, such as failing to properly handle daylight saving time conversion, or having timestamp errors when interacting with data across time zones.

[0032] Step 102: For any cluster, obtain multi-source operation and maintenance data related to failed test cases in the cluster.

[0033] Among them, multi-source operation and maintenance data may include, but are not limited to: test case details data, service operation log data, operation environment configuration information, program exception stack trace information, code change records corresponding to the target test task, relevant information of historical test cases, etc.

[0034] Step 103: Using a large model, based on multi-source operation and maintenance data, perform failure attribution analysis on failed test cases in the cluster to obtain the failure cause analysis results of the target test task; among which, the failure cause analysis results include test failure root cause type and attribution description information.

[0035] The root causes of test failures can include, but are not limited to, code change-related, environmental problem-related, and test case stability-related reasons.

[0036] The attribution description information can be a natural language explanation of why the corresponding test cases failed in the target test task.

[0037] As an example, multidimensional operation and maintenance data can be input into a large model to perform failure attribution analysis on failed test cases in the cluster through the large model, and the failure cause analysis results of the target test task can be obtained in response to the output of the large model.

[0038] The test anomaly attribution method of this disclosure clusters failed test cases corresponding to failed target test tasks to obtain at least one cluster; for any cluster, it acquires multi-source operation and maintenance data related to failed test cases in the cluster; using a large model, it performs failure attribution analysis on failed test cases in the cluster based on the multi-source operation and maintenance data to obtain the failure cause analysis results of the target test task; wherein, the failure cause analysis results include test failure root cause type and attribution description information. Therefore, clustering failed test cases and conducting unified attribution analysis on failed test cases within the same cluster can effectively avoid the problems of repetitive analysis and information fragmentation caused by independent investigation of individual test cases, which helps reduce the cost of manual troubleshooting of test failures. Then, by combining multi-source operation and maintenance data corresponding to failed test cases in the cluster for comprehensive attribution analysis and logical reasoning, standardized root cause types and clear and standardized attribution description information can be automatically extracted, improving the efficiency and accuracy of test failure attribution, reducing over-reliance on human experience, and realizing the automation and intelligence of test failure attribution. This facilitates relevant personnel to quickly locate problems and make repair decisions, shortens the test failure closed-loop cycle, and improves the overall operational efficiency of the R&D and testing process.

[0039] like Figure 2 As shown, this disclosure also proposes a method for testing anomaly attribution. Figure 2 As illustrated in the schematic diagram of the second embodiment of this disclosure, the test anomaly attribution method may include the following steps: Step 201: Obtain execution information corresponding to multiple failed test cases.

[0040] Among them, execution information can at least be used to characterize the interface information and assertion failure information during the execution process of the corresponding failed test cases.

[0041] The interface information may include, but is not limited to, the interface name, interface version, module to which the interface belongs, etc.

[0042] It should be noted that assertion failure information can be the information returned when the corresponding test case fails to execute, and this information can be status code assertions, field existence assertions, response time assertions, field value assertions, etc. For example, when the assertion failure information is a status code assertion, it can include the expected status code and the actual returned status code; when it is a field existence assertion, it can indicate whether a required field is missing from the response body; when it is a response time assertion, it can indicate whether the actual response time exceeds the expected threshold; when it is a field value assertion, it can indicate the specific situation where the actual value of a field is inconsistent with the expected value, and so on.

[0043] Step 202: Based on the execution information, cluster the multiple failed use cases to obtain at least one cluster.

[0044] As an example, based on the execution information, multiple failed test cases with the same interface name and assertion failure information can be identified as failing test cases belonging to the same cluster, thereby achieving clustering of multiple failed test cases and obtaining at least one cluster.

[0045] As another example, based on the execution information, multiple failed test cases with the same interface name, the same module to which the interface belongs, and the same assertion failure information can be identified as failed test cases belonging to the same cluster, so as to achieve clustering of multiple failed test cases and obtain at least one cluster.

[0046] Step 203: For any cluster, obtain multi-source operation and maintenance data related to failed test cases in the cluster.

[0047] Step 204: Using a large model, based on multi-source operation and maintenance data, perform failure attribution analysis on failed test cases in the cluster to obtain the failure cause analysis results of the target test task.

[0048] The failure cause analysis results include the root cause type of test failure and attribution description information.

[0049] It should be noted that the execution process of steps 203 to 204 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0050] Optionally, in any embodiment of this disclosure, the tag information of failed use cases is obtained; wherein the tag information is determined based on the corresponding failure cause analysis results; and the tag information of failed use cases is stored.

[0051] For example, in one application scenario, after the failure cause analysis results are displayed to relevant users in a visual manner, users can adopt or modify the failure cause analysis results. Then, the implementing entity of this disclosure can generate tag information for the corresponding failed test cases in the target test task based on the adopted or modified failure cause analysis results, and store the tag information of the failed test cases for subsequent data application.

[0052] Therefore, based on the root cause analysis results of test failures, accurate labeling information can be assigned to failed test cases and structured storage can be completed, so that each failed test case is bound to the corresponding root cause type and attribution characteristics. In this way, not only can the standardized classification and management of failed test cases be realized, which facilitates the rapid retrieval and statistical analysis of the frequency and distribution characteristics of various test anomalies, but it can also provide data support for historical failure review and test case optimization. At the same time, the labeled and stored failed test case data can be used as incremental training samples for large models, continuously optimizing the model's attribution analysis capabilities, realizing data closure, and further improving the intelligence and sustainable optimization capabilities of test anomaly attribution.

[0053] In one application scenario, the test anomaly method disclosed herein is applied to map software (also known as map engine or map service) testing. Multiple failed test cases can be clustered based on their corresponding interface names, the modules to which the interfaces belong, their geographic information, and assertion failure information, resulting in at least one cluster. For example, failed test cases with the same interface name, module, geographic information, and assertion failure information can be aggregated into one cluster. Then, multi-source operation and maintenance data of the failed test cases in any cluster in the map domain can be obtained. Finally, a large model is used to perform failure attribution analysis on the failed test cases within the cluster based on the aforementioned multi-source operation and maintenance data in the map domain, yielding the failure cause analysis results for the map software testing task.

[0054] Among them, geographical information can be used to indicate the geographical area, city, administrative region or tile coverage area corresponding to the map software test, including but not limited to city code, latitude and longitude range, area identifier, and geographical range corresponding to the base map version.

[0055] Understandably, map-related services are highly geographically dependent. Map data, road network information, service nodes, and operating environments may differ across different geographical regions. Therefore, clustering failed test cases based on geographical information can more accurately locate test anomalies related to geographical regions, improving the accuracy and relevance of subsequent failure attribution analysis.

[0056] The test anomaly attribution method of this disclosure obtains execution information corresponding to multiple failed test cases. The execution information at least characterizes the interface information and assertion failure information during the execution of the corresponding failed test cases. Based on the execution information, the multiple failed test cases are clustered to obtain at least one cluster. This enables the differentiation of multiple failed test cases, improves the accuracy and effectiveness of failed test case clustering, achieves centralized collection and unified analysis of similar faults, effectively reduces redundant troubleshooting work, avoids the inefficiency of analyzing each test case individually, and lays the foundation for subsequent centralized and targeted failure attribution analysis, thereby improving the overall efficiency and reliability of test anomaly analysis.

[0057] like Figure 3 As shown, this disclosure also proposes a method for testing anomaly attribution. Figure 3 As illustrated in the schematic diagram of the third embodiment of this disclosure, the test anomaly attribution method may include the following steps: Step 301: Cluster the failed test cases corresponding to the failed target test tasks to obtain at least one cluster.

[0058] Step 302: For any cluster, obtain multi-source operation and maintenance data related to failed test cases in the cluster.

[0059] It should be noted that the execution process of steps 301 to 302 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0060] Step 303: Based on multi-source operation and maintenance data, feature construction is performed to obtain the second feature.

[0061] Optionally, in any embodiment of this disclosure, when the multi-source operation and maintenance data includes code change records of the code to be released for testing by the target test task, the code attribute information of the code to be released is determined based on the code change records; a base code matching the code to be released is obtained based on the code attribute information; the code to be released and the base code are compared to determine the code change content of the code to be released relative to the base code; and a second feature is determined based on the code change content.

[0062] The code change log can be used to record the modifications made to the code to be released relative to historical versions, including but not limited to the modification time, the person who made the modification, and details of the code modifications.

[0063] The code attribute information can be used to indicate whether the code to be released is the main branch code. The main branch code refers to the code corresponding to the branch that carries the online stable version and the core baseline version in a software development project. It is the core main line of the entire code repository. All branches for feature development and defect fixing will eventually be merged into the main branch, and its code status directly reflects the code version in the online production environment (or the code that will be released to the production environment).

[0064] The baseline code can be used to compare with the code to be released in order to determine the code changes made by the code to be released relative to the baseline code.

[0065] In this embodiment of the disclosure, code change records can be analyzed to determine the code attribute information of the code to be released.

[0066] In order to obtain a base code that matches the code to be released based on code attribute information, optionally, in some embodiments of this disclosure, in response to the code attribute information indicating that the code to be released is a trunk branch code, the previous code version of the trunk branch corresponding to the code to be released can be determined as the base code; in response to the code attribute information indicating that the code to be released is a non-trunk branch code, the latest code version of the trunk branch corresponding to the code to be released can be determined as the base code.

[0067] In this context, the previous code version of the trunk branch can refer to the code corresponding to the last successfully released version of the trunk branch.

[0068] Therefore, in this embodiment of the disclosure, after determining the base code, the code to be released and the base code can be compared to determine the code changes of the code to be released relative to the base code; and a second feature can be determined based on the code changes, for example, the code changes can be directly determined as the second feature.

[0069] Therefore, differentiated baseline code matching strategies can be formulated for different branch attributes of the code to be released (i.e., trunk branch and non-trunk branch), ensuring the rationality and adaptability of the baseline code selection. Specifically, when the code to be released is trunk branch code, using the previous version of the trunk branch code as the baseline code accurately focuses on the code changes in this trunk iteration, avoiding interference from irrelevant historical versions. When the code to be released is non-trunk branch code, using the latest version of the trunk branch code as the baseline code effectively aligns the differences between the branch code and the trunk baseline, fully identifying the branch-specific code changes. This scenario-based baseline code matching method effectively improves the accuracy and completeness of code change identification, enabling the second feature constructed based on this to accurately reflect the correlation between test failures and code changes. This provides more accurate and reliable feature support for subsequent failure attribution analysis of large models, further improving the efficiency and accuracy of root cause localization of test anomalies.

[0070] It should be noted that by fully utilizing the code change records corresponding to the target test task, extracting code attribute information, matching benchmark code, and comparing differences, the code change content of the code to be released can be accurately located. This allows the construction of a second feature, which can quantify and structure the code-level change information into the feature representation. This makes the second feature more closely related to the intrinsic relationship between test failures and code changes, providing the large model with more targeted input information that is closer to the root cause of the failure. This effectively improves the accuracy and interpretability of root cause analysis, reduces the probability of missed or false positives in test anomalies caused by code changes, and further improves the reliability and intelligence level of test failure attribution.

[0071] Optionally, in some embodiments of this disclosure, when the multi-source operation and maintenance data includes service operation log data and / or program exception stack trace information, a large model can be used to extract information from the service operation log data and / or program exception stack trace information to obtain key information related to test failure; the key information is then identified as the second feature.

[0072] Among them, key information can be used to indicate the key error data that caused the test case to fail, the program to be abnormal, or the function to be abnormal during the execution of the corresponding test case. For example, key information may include high-importance log information such as warnings and fatalities in the service operation log that have fault attribution value, as well as the exception type, the location of the exception, and the exception call chain in the program exception stack trace information.

[0073] For example, service log data and program exception stack trace information can be input into an LLM (Large Language Model) to leverage the powerful contextual understanding capabilities of the LLM to filter noise and extract information from the service log data and program exception stack trace information, thereby obtaining structured key information, which can then be identified as the second feature.

[0074] Therefore, it is possible to fully utilize fine-grained, high-value runtime anomaly data such as service operation logs and program exception stack traces. Through large-scale models, log and stack information can be automatically parsed and key information extracted to extract key information directly related to test failures and structure it into a second feature. This approach eliminates the need for manual log rule compilation or parsing script writing, reducing the complexity and cost of feature construction while improving the completeness and accuracy of key information extraction. This allows the second feature to more intuitively and comprehensively reflect the on-site anomaly information of test failures, providing a solid and reliable feature foundation for subsequent accurate root cause analysis by large-scale models, and further improving the automation and accuracy of test anomaly attribution.

[0075] Optionally, in some embodiments of this disclosure, when the multi-source operation and maintenance data includes operating environment information, feature extraction is performed on the operating environment information to obtain environmental features; wherein, the environmental features include environmental indicator values ​​and / or environmental configuration data in multiple dimensions; and a second feature is determined based on the environmental features.

[0076] The environmental metrics values ​​across multiple dimensions may include, but are not limited to: runtime metrics such as CPU utilization, memory usage, and network latency of the CI / CD pipeline service environment; and environmental configuration data may include, but are not limited to: environment type, deployment version, dependency package version, network configuration, timeout configuration, service address configuration, database connection configuration, etc.

[0077] In this embodiment of the disclosure, feature extraction can be performed on the runtime environment information to obtain environmental features, and a second feature can be determined based on the environmental features. For example, a feature extraction model can be used to extract features from the runtime environment information, and the extracted environmental features can be determined as the second feature. It should be noted that the feature extraction model can be a deep learning model, a machine learning model, etc., and this disclosure does not limit it.

[0078] Therefore, multi-dimensional environmental indicator values ​​and environmental configuration data in the operating environment can be transformed into standardized environmental features and integrated into the second feature. This fully considers the impact of differences in the operating environment on the test task execution results and avoids root cause analysis bias caused by ignoring environmental factors. By extracting structured features from the operating environment information, the second feature can more comprehensively cover the potential influencing factors of test failure, improve the completeness and representativeness of the features, provide richer and more objective judgment basis for the large model, thereby improving the accuracy and robustness of test failure attribution and further ensuring the comprehensiveness and reliability of fault root cause localization.

[0079] Step 304: Using a large model, based on the second feature, determine the failure cause analysis results of the target test task.

[0080] It should be noted that the explanation of the failure cause analysis results in step 103 also applies to this embodiment, and will not be repeated here.

[0081] As an example, the second feature can be input into a large model, and the failure analysis results of the target test task can be obtained in response to the output of the large model.

[0082] As another example, the prompt words can be determined based on the second feature; the prompt words are then input into a large model, and the failure analysis results of the target test task are obtained in response to the output of the large model.

[0083] For example, the second feature can be filled into the corresponding position in the prompt word template to obtain the prompt word, and the prompt word can be input into the large model. In response to the output of the large model, the failure reason analysis results of the target test task can be obtained.

[0084] Therefore, targeted prompts can be automatically constructed based on the second feature, transforming structured and standardized feature information into input content that the large model can directly understand. Furthermore, by inputting the constructed prompts into the large model and obtaining the model output, the semantic understanding and logical reasoning capabilities of the large model can be fully utilized to achieve automated analysis and output of test failure reasons. This can effectively improve the accuracy, consistency, and readability of the large model's output results, making it easier to quickly obtain clear and reliable failure reason analysis results, and further improve the intelligence level and analysis efficiency of test anomaly attribution.

[0085] To obtain the prompt words, optionally, in some embodiments of this disclosure, the target database is queried based on the use case metadata of failed use cases in the cluster to obtain historical test failure information of the failed use cases; and the prompt words are determined based on the second feature and the historical test failure information.

[0086] The test case metadata may include, but is not limited to: the test case identifier, test case name, interface name, and interface path of the corresponding test case; the test task identifier, version number, and module name to which the corresponding test case belongs; and the execution environment and execution time of the corresponding test case, etc. The test case identifier can be used to uniquely identify the corresponding test case, and the test task identifier can be used to uniquely identify the test task.

[0087] The target database can be a database that stores historical test data, historical failure records, and test case execution records.

[0088] Historical test failure information may include, but is not limited to: historical failure time, historical failure count, historical failure reason, historical error log, historical stack information, historical associated code change information, historical error code information, etc.

[0089] Among them, historical stack information is structured data that reflects the program's running state when a test case fails. It can include key information such as all function or method call paths, line numbers, process or thread identifiers, and memory addresses traced back from the failure trigger point.

[0090] Among them, historical associated code change information can be all code modification records related to the functional modules and code segments covered by the test case within the time window of test case execution failure. This can include, but is not limited to, the submitter of the code change, the submission time, the path of the changed file, the specific code additions, deletions and modifications, the version number, the change description (such as the fixed vulnerability or the added function), and the associated requirement or defect number.

[0091] Among them, historical error code information can be standardized character or numeric codes that are output according to preset error coding rules when the corresponding test case fails to execute, which are used to identify the specific failure type, as well as the corresponding text description, fault level, associated solutions and other supporting information.

[0092] For example, the target database can be queried based on the test case identifier of the failed test case to obtain the historical test failure information of the failed test case; then, the second feature and the historical test failure test case can be filled into the corresponding position in the prompt word template to obtain the prompt word.

[0093] Therefore, by querying historical test failure information obtained from the target database, the correlation between the current failure and historical failures can be effectively explored. This allows the prompt words to not only contain real-time feature information of the current test failure but also incorporate the contextual background of historical failures. This helps the large model to assist in root cause judgment from the perspective of historical experience, avoiding repeated pitfalls. This prompt word construction method, which integrates real-time features and historical information, can provide the large model with more comprehensive and valuable analytical basis, improve the accuracy and depth of the failure cause analysis results output by the large model, and reduce attribution bias caused by one-sided information, further enhancing the comprehensiveness and reliability of test anomaly attribution.

[0094] Optionally, in any embodiment of this disclosure, after obtaining the code changes of the code to be released relative to the base code, a large model can be used to analyze the correlation between the code changes and the failed test cases in the cluster, and obtain the correlation analysis results of the target test task.

[0095] Among them, the results of the association analysis can be used to indicate whether failed use cases in a cluster are associated with code changes.

[0096] As an example, code changes and failed use cases in a cluster can be input into a large model to analyze the correlation between the code changes and failed use cases in the cluster. The correlation analysis results can be obtained in response to the output of the large model.

[0097] Therefore, by leveraging the semantic understanding and logical reasoning capabilities of large models, the correlation between failed test cases and code changes can be identified, and a clear "correlation" judgment result can be output. This approach can help quickly filter out test failure scenarios caused by code changes, eliminate interference from irrelevant factors, and help relevant personnel focus on the core problem source. At the same time, the correlation judgment result can define a clear scope for subsequent root cause analysis, avoid ineffective investigation, improve the pertinence and efficiency of test anomaly attribution, and help further shorten the fault location cycle and reduce R&D testing costs.

[0098] Optionally, in some embodiments of this disclosure, code change domain information is determined based on the code change content; use case target domain information is determined based on use case details data of failed use cases in the cluster; and a large model is used to determine the correlation analysis results based on the code change domain information and the use case target domain information.

[0099] Code change field information can be used to indicate the modules, interfaces, classes, parameter configurations, etc., involved in the code changes. In other words, code change field information can be used to clearly identify the modules that have been modified, the interfaces that have been changed, the code of the classes that have been adjusted, the parameter configurations that have been modified, etc.

[0100] The target domain information of a test case can be used to indicate the functions, interfaces, modules, etc., covered by the corresponding failed test case. In other words, the target domain information of a test case can be used to clarify the functions, interfaces, modules, etc., that the corresponding failed test case was originally intended to verify.

[0101] As an example, code change details can be input into a large model to determine code change domain information; use case details of failed use cases in a cluster can be input into the large model to determine use case target domain information; furthermore, code change domain information and use case target domain information can be input into the large model to analyze the above information, and then the correlation analysis results can be obtained based on the output of the large model.

[0102] Therefore, the code change content and failed test case details can be extracted into structured code change domain information and test case target domain information, respectively, to accurately define the "scope of code modification" and the "scope of test case execution". Then, based on the large model, targeted correlation analysis of these two types of domain information can be performed to accurately determine whether failed test cases are related to code changes, avoiding misjudgments caused by text matching alone. This approach of defining the domain first and then analyzing improves the accuracy and interpretability of the correlation analysis results, helping relevant personnel to quickly locate the business function domain to which the fault belongs, further narrowing the scope of root cause investigation, and improving the efficiency and accuracy of test anomaly attribution.

[0103] Optionally, in some embodiments of this disclosure, code change domain information and use case target domain information are input into a large model to determine whether there is an intersection between the code change domain information and the use case target domain information in multiple target dimensions; in response to the existence of an intersection between the code change domain information and the use case target domain information in multiple target dimensions, the association analysis result is determined to be the association between failed use cases and code change content in the cluster.

[0104] The target dimension can include, but is not limited to, modules, interfaces, logic, etc.

[0105] For example, code change domain information and use case target domain information can be input into a large model. The large model analyzes the intersection between the code change domain information and the use case target domain information to determine whether there is an intersection between the code change domain information and the use case target domain information at the module, interface, and logic levels. If there is an intersection between the code change domain information and the use case target domain information at the module, interface, and logic levels, the association analysis result is determined to be an association between the failed use cases in the cluster and the code change content.

[0106] It's important to note that test failures may be due to environmental issues. In such cases, if the code change domain information and the test case target domain information overlap at the module, interface, or logic level, the correlation analysis result indicates that the failed test case in the cluster is associated with the code change content. For example, if the code change content is the "user login module," its corresponding code change domain information is the login module, and the failed test case is the "login interface test case," even if the log information shows that the failure is due to a database timeout in the test environment, and the direct cause of the failure is the environment, the fact that the code change domain information of the code change content and the test case target domain information of the failed test case overlap at the module level indicates that the failed test case is associated with the code change content.

[0107] Therefore, the correlation between failed use cases and code changes can be accurately determined from multiple target dimensions.

[0108] Optionally, in some embodiments of this disclosure, when the use case target domain information indicates the function covered by the corresponding failed use case, if there is no intersection between the function covered by the failed use case in the cluster and the code change domain information, then the association analysis result is determined to be that the failed use case in the cluster is not associated with the code change content.

[0109] Optionally, in some embodiments of this disclosure, a prompt message is generated based on the failure cause analysis results and / or correlation analysis results; and the prompt message is sent to the target user.

[0110] As an example, a prompt message can be generated based on the results of the failure cause analysis and sent to the target user.

[0111] As another example, a prompt message can be generated based on the results of failure cause analysis and correlation analysis, and then sent to the target user.

[0112] Therefore, by generating and sending messages, users can promptly understand the results of failure cause analysis and / or correlation analysis, enabling relevant personnel to carry out fault repair and problem handling in a timely manner, further shortening the test anomaly closed-loop cycle, and improving the collaborative efficiency and automation level of the R&D and testing processes.

[0113] To acquire target users, optionally, in some embodiments of this disclosure, the process of determining target users includes the following steps: when the test failure root cause type indicated by the failure analysis result is code change-introduced or the association analysis result indicates that the failed test cases in the cluster are associated with the code change content, the code submitter of the code to be released can be determined as the target user; when the test failure root cause type indicated by the failure analysis result is not code change-introduced and the association analysis result indicates that the failed test cases in the cluster are not associated with the code change content, the currently on-duty tester can be determined as the target user.

[0114] Therefore, based on the root cause type of test failure and the results of correlation analysis, it is possible to accurately distinguish and intelligently match target users, achieve precise delivery of fault notifications and assign responsibility to individuals, avoid invalid notifications and cross-role shirking, effectively improve the efficiency of anomaly response, shorten the fault handling chain, and further improve the intelligence and collaborative efficiency of the R&D testing process.

[0115] The test anomaly attribution method of this disclosure constructs a second feature based on multi-source operation and maintenance data; then, using a large model, based on the second feature, it determines the failure cause analysis results of the target test task. This fully integrates key information from multi-source operation and maintenance data, improves the standardization and usability of the data through structured feature construction, and facilitates a more efficient and accurate understanding and reasoning of the test task failure causes by the large model. Using a large model for failure cause analysis based on the second feature fully leverages the semantic understanding and logical reasoning capabilities of the large model, improving the accuracy and comprehensiveness of the output failure cause analysis results, achieving automation and intelligence in test failure cause analysis, thereby helping to improve fault location efficiency, shorten fault repair cycles, and enhance the stability and efficiency of the overall testing and development process.

[0116] To clearly illustrate the test anomaly attribution method disclosed in this publication, a detailed explanation is provided below with examples.

[0117] As an example, the test anomaly attribution method disclosed herein is applied to an LLM-based intelligent attribution agent for continuous integration test case failures. Figure 4This is a schematic diagram of the Agent's structure, such as... Figure 4 As shown, the Agent may include an event receiving module 401, a use case aggregation module 402, a multimodal feature construction module 403, an LLM intelligent attribution engine 404, an output and responsibility notification module 405, and a data management module 406, wherein: 1. Event receiving module 401 is used to receive test failure event notifications from the target platform (such as CI / CD platform, etc.), then parse the notification content to extract the failed tasks containing basic test task information; then, store these failed tasks in the task database in real time, and mark these failed tasks in the task database and mark their status as pending analysis, so that the failed tasks to be analyzed can be identified based on the task status later. The basic information of the test task includes, but is not limited to: test task identifier, code repository information, etc.

[0118] 2. The test case aggregation module 402 is used to pull relevant data of the failed tasks to be analyzed (referred to as target test tasks in this disclosure) from the task database. Then, based on these data, the failed test cases corresponding to the failed tasks to be analyzed are clustered to obtain at least one cluster.

[0119] The relevant data for the failed tasks to be analyzed may include use case details data, which may include use case metadata (such as use case identifiers).

[0120] For example, data analysis algorithms and clustering models can be used to cluster the corresponding failed test cases based on the relevant data of the failed tasks to be analyzed, so as to obtain at least one cluster.

[0121] 3. Multimodal feature construction module 403 is used to acquire the original heterogeneous data related to the failure of the failed task to be analyzed (referred to as multidimensional operation and maintenance data in this disclosure), and to construct features based on the original heterogeneous data to obtain multimodal features.

[0122] The original heterogeneous data may include, but is not limited to: test case details of failed test cases, service operation log data, operation environment configuration information, program exception stack trace information, code change records corresponding to the failed tasks to be analyzed, relevant information of historical test cases, etc.

[0123] To acquire multimodal features, the multimodal feature construction module 403 can also be used, for example, to: 1) When the code to be released used for testing in the failed task to be analyzed is trunk branch code, since the commit history of the code to be released is consistent with the latest commit history of the trunk, the commit history corresponding to the last successful release version of the trunk code (referred to as the previous code version of the trunk branch in this disclosure) can be used as the baseline code. Then, the code to be released is compared with the baseline code to obtain the incremental changes of the code to be released relative to the baseline code (referred to as the code change content in this disclosure). When the code to be released used for testing in the failed task to be analyzed is non-trunk branch code, since the commit history of the code to be released is different from the latest commit history of the trunk, the multimodal feature construction module can be used to use the latest commit history of the trunk as the baseline code. Then, the code to be released is compared with the baseline code to obtain the incremental changes of the code to be released relative to the baseline code.

[0124] 2) Extract information from the service runtime log data and program exception stack trace information during the test run. The context understanding capability of LLM can be used to filter noise and extract key error information from the service runtime log data and program exception stack trace information, and obtain structured key evidence (referred to as key information in this disclosure).

[0125] 3) Obtain CI / CD pipeline service environment indicators and environment configuration data as environment characteristics; 4) Use RAG (Retrieval-augmented Generation) to retrieve historical instability data (referred to as historical test failure information in this disclosure) and module expert knowledge (such as error codes) of test cases in the knowledge base.

[0126] Historical instability data can include historical failure times, historical failure counts, historical failure reasons, historical error logs, historical stack traces, historical associated code change information, and so on.

[0127] 5) Multimodal features can be constructed based on incremental changes, key evidence, environmental characteristics, historical instability data, and module expert knowledge. These multimodal features can be structured into prompt words to be provided to the LLM intelligent attribution engine 404.

[0128] IV. The LLM intelligent attribution engine 404 is used to receive prompt words provided by the multimodal feature construction module, utilize the cross-modal semantic understanding and causal reasoning capabilities of LLM to attribute test failures, and can obtain output results based on LLM. The output results may include failure cause analysis results and correlation determination conclusions between failed test cases and incremental changes (referred to as correlation analysis results in this disclosure). The failure cause analysis results may include the fault type (referred to as test failure attribution type in this disclosure), the key evidence chain supporting the conclusion, and the intelligent attribution conclusion of the analysis process (referred to as attribution description information in this disclosure).

[0129] For example, the LLM Smart Attribution Engine 404 can be used for: 1. Using LLM, based on input prompts, perform cross-modal causal reasoning to determine the direct root cause of test failure, i.e., perform the following separately: Intelligent attribution classification: Fault types, including but not limited to: code change-related, environmental problem-related, use case stability-related, etc.; Evidence chain extraction: LLM is used to extract and structure key evidence chains that support the attribution conclusions from lengthy unstructured data, such as identifying specific anomaly types, error codes, key variable values, and linking them to the most recent modification of the environment configuration file.

[0130] 2. Utilize the semantic understanding capabilities of LLM to determine the potential impact and correlation between failed use cases and incremental changes.

[0131] For example, when performing correlation determination, the LLM intelligent attribution engine 404 can make judgments based on use case target domain information, code change domain information, and dependencies. When there is an intersection between the code change domain information and the use case target domain information in terms of modules, interfaces, or logic, it can be determined that the failed use case is associated with the incremental change content. When the use case target domain information indicates the function covered by the corresponding failed use case, if there is no intersection between the function covered by the failed use case and the code change domain information, it is determined that the failed use case is not associated with the incremental change content.

[0132] Among them, code change field information can be used to indicate the modules, interfaces, classes, parameter configurations, etc. involved in the code changes; Among them, the target domain information of the test case can be used to indicate the functions, interfaces, modules, etc. covered by the corresponding failed test cases; The dependencies can be inferred using LLM based on failed use cases and incremental changes.

[0133] 5. Output and Responsibility Notification Module 405 is used to convert the conclusions output by the LLM attribution engine into user-friendly attribution reports. This module can also automatically identify the primary responsible party (referred to as the target user in this disclosure) based on the intelligent attribution conclusions, and push the attribution report and the information of the primary responsible party to the relevant development and testing personnel's collaboration groups in real time through channels such as API (Application Programming Interface), instant messaging, or email.

[0134] For example, when the fault type is code change introduced or the correlation determination conclusion indicates that the failed test case is associated with the incremental change, the code commit record can be automatically associated and the code change committer can be identified as the first responsible person; when the fault type is not code change introduced and the correlation determination conclusion indicates that the failed test case is not associated with the incremental change, the currently on-duty tester can be identified as the first responsible person.

[0135] VI. Data management module 406 can be used to obtain fault tags (referred to as tag information in this disclosure), establish a correspondence between failed use cases and fault tags, and then store the failed use cases carrying fault tags in the knowledge base.

[0136] Optionally, fault labels can be determined based on attribution reports. For example, after receiving an attribution report, developers and testers can modify the attribution conclusions and submit manually modified attribution conclusions, which can then be identified as fault labels.

[0137] Optionally, failed use cases carrying fault labels can be used for RAG retrieval to continuously optimize the attribution accuracy of the Agent in relevant business scenarios.

[0138] In summary, this disclosed LLM-based intelligent attribution agent for continuous integration test case failures achieves automated, high-precision, and interpretable intelligent attribution and responsibility determination for failed test cases in continuous integration testing by introducing a large-scale language model as the core attribution engine. The agent first uses a failed test case clustering algorithm to aggregate failed test cases with similar characteristics. Next, it collects multi-source data on failed test cases, encompassing heterogeneous information such as test case details, code change records, service logs, service monitoring information, environment troubleshooting results, service database information, historical data, and expert knowledge. Then, leveraging the powerful cross-modal semantic understanding and causal reasoning capabilities of LLM, it deeply mines this heterogeneous data to extract key evidence chains. Based on these key evidence chains, it generates high-confidence intelligent attribution conclusions, clarifying the causes of failed test cases and the corresponding responsibility attribution. Finally, the generated intelligent attribution conclusions can be directly pushed to the relevant responsible parties, improving the efficiency and accuracy of handling failed test cases in continuous integration testing.

[0139] To implement the above embodiments, this disclosure also provides a device for testing anomaly attribution. For example... Figure 5 As shown, Figure 5 This is a schematic diagram according to the fourth embodiment of the present disclosure. The test anomaly attribution device 500 may include: a clustering module 501, a first acquisition module 502, and a first analysis module 503.

[0140] Among them, the clustering module 501 is used to cluster the failed test cases corresponding to the failed target test tasks to obtain at least one cluster.

[0141] The first acquisition module 502 is used to acquire multi-source operation and maintenance data related to failed test cases in any cluster.

[0142] The first analysis module 503 is used to perform failure attribution analysis on failed test cases in the cluster using a large model and multi-source operation and maintenance data, and obtain the failure cause analysis results of the target test task; among which, the failure cause analysis results include test failure attribution type and attribution description information.

[0143] In one possible implementation of this disclosure, there are multiple failed test cases. The clustering module 501 is used to: obtain execution information corresponding to the multiple failed test cases; wherein the execution information is used to characterize the interface information and assertion failure information during the execution process of the corresponding failed test cases; and cluster the multiple failed test cases according to the execution information to obtain at least one cluster.

[0144] In one possible implementation of this disclosure, the first analysis module 503 is used to: construct features based on multi-source operation and maintenance data to obtain a second feature; and use a large model to determine the failure cause analysis results of the target test task based on the second feature.

[0145] In one possible implementation of this disclosure, the multi-source operation and maintenance data includes code change records of the code to be released for testing by the target test task; a first analysis module 503 is used to: determine code attribute information of the code to be released based on the code change records; obtain a benchmark code that matches the code to be released based on the code attribute information; compare the code to be released and the benchmark code to determine the code change content of the code to be released relative to the benchmark code; and determine a second feature based on the code change content.

[0146] In one possible implementation of this disclosure, the test anomaly attribution device 500 may further include: The second analysis module is used to analyze the correlation between code changes and failed test cases in the cluster using a large model, and obtain the correlation analysis results of the target test task. The correlation analysis results are used to indicate whether the failed test cases in the cluster are related to the code changes.

[0147] In one possible implementation of this disclosure, the second analysis module is used to: determine code change domain information based on the code change content; determine use case target domain information based on use case detail data of failed use cases in the cluster; and determine the correlation analysis results using a large model based on the code change domain information and the use case target domain information.

[0148] In one possible implementation of this disclosure, the second analysis module is configured to: input code change domain information and use case target domain information into a large model to determine whether there is an intersection between the code change domain information and the use case target domain information in multiple target dimensions; and, in response to the existence of an intersection between the code change domain information and the use case target domain information in multiple target dimensions, determine that the association analysis result is that the failed use cases in the cluster are associated with the code change content.

[0149] In one possible implementation of this disclosure, the test anomaly attribution device 500 may further include: The generation module is used to generate prompt messages based on the results of failure cause analysis and / or correlation analysis.

[0150] The sending module is used to send notification messages to the target user.

[0151] In one possible implementation of this disclosure, the test anomaly attribution device 500 may further include: The determination module is used to: determine the code submitter of the code to be released as the target user in response to the test failure attribution type indicated by the failure analysis results or the association analysis results indicating that the failure test cases in the cluster are associated with the code change content; and determine the on-duty tester as the target user in response to the test failure attribution type indicated by the failure analysis results or the failure test cases in the cluster are not associated with the code change content.

[0152] In one possible implementation of this disclosure, the first analysis module 503 is configured to: in response to code attribute information indicating that the code to be released is a trunk branch code, determine the previous code version of the trunk branch corresponding to the code to be released as the base code; and in response to code attribute information indicating that the code to be released is a non-trunk branch code, determine the latest code version of the trunk branch corresponding to the code to be released as the base code.

[0153] In one possible implementation of this disclosure, the multi-source operation and maintenance data includes service operation log data and / or program exception stack trace information; the first analysis module 503 is used to: use a large model to extract information from the service operation log data and / or program exception stack trace information to obtain key information related to test failure; and determine the key information as a second feature.

[0154] In one possible implementation of this disclosure, the multi-source operation and maintenance data includes operating environment information. The first analysis module 503 is used to: extract features from the operating environment information to obtain environmental features; wherein the environmental features include multiple first-dimensional environmental indicator values ​​and / or environmental configuration data; and determine a second feature based on the environmental features.

[0155] In one possible implementation of this disclosure, the first analysis module 503 is configured to: determine a prompt word based on a second feature; input the prompt word into a large model, and in response to the output of the large model, obtain the failure reason analysis result of the target test task.

[0156] In one possible implementation of this disclosure, the first analysis module 503 is configured to: query the target database based on the test case metadata of failed test cases in the cluster to obtain historical test failure information of failed test cases; and determine prompt words based on the second feature and the historical test failure information.

[0157] In one possible implementation of this disclosure, the test anomaly attribution device 500 may further include: The second acquisition module is used to acquire the tag information of failed test cases; the tag information is determined based on the corresponding failure cause analysis results.

[0158] The storage module is used to store the tag information of failed test cases.

[0159] It should be noted that the test anomaly attribution device provided in this embodiment can achieve the above-mentioned... Figures 1 to 3 The method steps implemented in this embodiment of the test anomaly attribution method are the same as those in the method embodiment, and can achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail here.

[0160] The test anomaly attribution device of this disclosure clusters failed test cases corresponding to failed target test tasks to obtain at least one cluster; for any cluster, it acquires multi-source operation and maintenance data related to failed test cases in the cluster; using a large model, it performs failure attribution analysis on failed test cases in the cluster based on the multi-source operation and maintenance data to obtain the failure cause analysis results of the target test task; wherein, the failure cause analysis results include test failure root cause type and attribution description information. Therefore, clustering failed test cases and conducting unified attribution analysis on failed test cases within the same cluster can effectively avoid the problems of repetitive analysis and information fragmentation caused by independent investigation of individual test cases, which helps reduce the cost of manual troubleshooting of test failures. Then, by combining multi-source operation and maintenance data corresponding to failed test cases in the cluster for comprehensive attribution analysis and logical reasoning, standardized root cause types and clear and standardized attribution description information can be automatically extracted, improving the efficiency and accuracy of test failure attribution, reducing over-reliance on human experience, and realizing the automation and intelligence of test failure attribution. This facilitates relevant personnel to quickly locate problems and make repair decisions, shortens the test failure closed-loop cycle, and improves the overall operational efficiency of the R&D and testing process.

[0161] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, are all carried out with the user's consent and comply with relevant laws and regulations, and do not violate public order and good morals.

[0162] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0163] Figure 6A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0164] like Figure 6 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0165] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0166] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the test anomaly attribution method. For example, in some embodiments, the test anomaly attribution method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the test anomaly attribution method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the test anomaly attribution method by any other suitable means (e.g., by means of firmware).

[0167] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0168] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0169] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0171] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0172] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0173] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0174] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for testing anomaly attribution, wherein, The method includes: Cluster the failed test cases corresponding to the failed target test tasks to obtain at least one cluster; For any of the clusters, obtain multi-source operation and maintenance data related to failed test cases in the cluster; Using a large model, based on the multi-source operation and maintenance data, failure attribution analysis is performed on the failed test cases in the cluster to obtain the failure cause analysis results of the target test task; wherein, the failure cause analysis results include test failure attribution type and attribution description information.

2. The method according to claim 1, wherein, The number of failed test cases is multiple. The clustering of failed test cases corresponding to the failed target test task yields at least one cluster, including: Obtain execution information corresponding to multiple failed test cases; wherein, the execution information is at least used to characterize the interface information and assertion failure information during the execution process of the corresponding failed test cases; Based on the execution information, the multiple failed use cases are clustered to obtain at least one cluster.

3. The method according to any one of claims 1 or 2, wherein, The method employs a large model and, based on the multi-source operation and maintenance data, performs failure attribution analysis on failed test cases within the cluster to obtain the failure cause analysis results of the target test task, including: Based on the multi-source operation and maintenance data, a second feature is obtained by feature construction. Using the large model, based on the second feature, the failure cause analysis results of the target test task are determined.

4. The method according to claim 3, wherein, The multi-source operation and maintenance data includes the code change records of the release code used for testing by the target test task; The second feature is obtained by constructing features based on the multi-source operation and maintenance data, including: Based on the code change records, the code attribute information of the code to be released is determined; Based on the code attribute information, obtain the baseline code that matches the code to be released; The code to be released is compared with the benchmark code to determine the code changes made by the code to be released relative to the benchmark code. Based on the code change, the second feature is determined.

5. The method according to claim 4, wherein, The method further includes: Using the large model, the correlation between the code changes and the failed test cases in the cluster is analyzed to obtain the correlation analysis results of the target test task; The correlation analysis results are used to indicate whether the failed use cases in the cluster are associated with the code change content.

6. The method according to claim 5, wherein, The large model is used to analyze the correlation between the code changes and the failed test cases in the cluster, and the correlation analysis results of the target test task are obtained, including: Based on the code changes, determine the code change domain information; Based on the use case details data of failed use cases in the cluster, the target domain information of the use cases is determined; Using the large model, the correlation analysis results are determined based on the code change domain information and the use case target domain information.

7. The method according to claim 6, wherein, The process of using the large model, based on the code change domain information and the use case target domain information, to determine the correlation analysis results includes: The code change domain information and the use case target domain information are input into the large model to determine whether there is any intersection between the code change domain information and the use case target domain information in multiple target dimensions; If the code change domain information and the use case target domain information have an intersection in multiple target dimensions, then the association analysis result is determined to be that the failed use cases in the cluster are associated with the code change content.

8. The method according to claim 5, wherein, The method further includes: Based on the failure cause analysis results and / or the correlation analysis results, a prompt message is generated; Send the notification message to the target user.

9. The method according to claim 8, wherein, The process of determining the target user includes the following steps: In response to the failure analysis result indicating that the test failure attribution type is code change introduced or the association analysis result indicates that the failed test cases in the cluster are associated with the code change content, the code submitter of the code to be released is identified as the target user. In response to the failure analysis result indicating that the test failure attribution type is non-code change introduced, and the association analysis result indicating that the failed test cases in the cluster are not associated with the code change content, the on-duty tester is identified as the target user.

10. The method according to claim 4, wherein, The step of obtaining the baseline code matching the code to be released based on the code attribute information includes: In response to the code attribute information indicating that the code to be released is trunk branch code, the previous code version of the trunk branch corresponding to the code to be released is determined as the base code; In response to the code attribute information indicating that the code to be released is a non-main branch code, the latest code version of the main branch corresponding to the code to be released is determined as the base code.

11. The method according to claim 3, wherein, The multi-source operation and maintenance data includes service operation log data and / or program exception stack trace information; the feature construction of the multi-source operation and maintenance data to obtain the second feature includes: Using the large model, information is extracted from the service operation log data and / or the program exception stack trace information to obtain key information related to test failure; The key information is identified as the second feature.

12. The method according to claim 3, wherein, The multi-source operation and maintenance data includes operating environment information. The step of constructing features from the multi-source operation and maintenance data to obtain a second feature includes: Feature extraction is performed on the operating environment information to obtain environmental features; wherein, the environmental features include multiple first-dimensional environmental indicator values ​​and / or environmental configuration data; The second feature is determined based on the environmental characteristics.

13. The method according to claim 3, wherein, The step of using the large model and determining the failure cause analysis results of the target test task based on the second feature includes: Based on the second feature, the prompt word is determined; The prompt words are input into the large model, and in response to the output of the large model, the failure reason analysis results of the target test task are obtained.

14. The method according to claim 13, wherein, The step of determining prompt words based on the second feature includes: Based on the test case metadata of the failed test cases in the cluster, the target database is queried to obtain the historical test failure information of the failed test cases; The prompt word is determined based on the second feature and the historical test failure information.

15. The method according to any one of claims 1 or 2, wherein, The method further includes: Obtain the tag information of the failed test cases; wherein the tag information is determined based on the corresponding failure cause analysis results; The tag information of the failed test cases is stored.

16. A device for testing anomaly attribution, wherein, The device includes: The clustering module is used to cluster the failed test cases corresponding to the target test tasks that failed, and obtain at least one cluster. The first acquisition module is used to acquire multi-source operation and maintenance data related to failed test cases in any of the clusters. The first analysis module is used to perform failure attribution analysis on failed test cases in the cluster based on the multi-source operation and maintenance data using a large model, and to obtain the failure cause analysis results of the target test task; wherein, the failure cause analysis results include test failure attribution type and attribution description information.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images